mirror of
https://github.com/alirezarezvani/claude-skills.git
synced 2026-08-28 04:24:58 +00:00
Merge pull request #993 from alirezarezvani/claude/spinning-up-book-skill-hhbjpy
This commit is contained in:
Alireza Rezvani
GitHub
commit
8def1c40ec
40 changed files with 3097 additions and 49 deletions
|
|
@ -8,7 +8,7 @@
|
|||
"homepage": "https://github.com/alirezarezvani/claude-skills",
|
||||
"repository": "https://github.com/alirezarezvani/claude-skills",
|
||||
"metadata": {
|
||||
"description": "387 production-ready skills across 20 domains (engineering, engineering-core, marketing, product, c-level, c-level-agents, compliance-os, project management, RA/QM, business growth, finance, productivity, marketing top-level, research, research-ops, business-operations, commercial, markdown-html, loop-library, plus standards). 727 Python tools, 842 reference guides, 117 agents (cs-* + personas), 149 slash commands across 98 marketplace plugins. v2.11.2 vendors engineering/skillopt-sleep — a verbatim copy of microsoft/SkillOpt's stdlib-only skillopt_sleep engine + Claude Code plugin surface, giving a local agent a nightly gated self-improvement cycle (read-only session harvest -> mine -> offline replay -> held-out-gated CLAUDE.md/SKILL.md edits -> staged for explicit /skillopt-sleep adopt). productivity/fable-goal (unreleased, post-v2.11.1) converts a rambling description of a desired outcome into one polished /goal prompt for a fresh autonomous session. v2.11.1 turns product-team and project-management into agent-harness domains: fork-orchestrators with deterministic goal routers, a Jira MCP snapshot bridge (Kanban flow metrics + Monte Carlo forecasting), a delegation-governance loop gate, a continuous-discovery cadence tracker, and an Opportunity Solution Tree linter, with /cs:pm and /cs:product command families. v2.10.3 completes the markdown-html domain with md-slides — slide-deck converter (arrow-key / Space / PgDn / Home/End / P keyboard navigation + presenter mode with split-view clock + speaker notes + next-slide preview + URL-hash deep linking like #3 for direct slide jumps + @media print page-per-slide for browser-native PDF export). Reuses md-document's markdown parser; vanilla JS only (no framework runtime); Prism.js opt-in via --syntax. Joins md-review (v2.10.2 code-review converter), md-document (v2.10.1 long-form converter), and the v2.10.0 foundation (orchestrator + design-system). Compatible with Claude Code, Codex CLI, Gemini CLI, Cursor, OpenClaw, Hermes Agent, Mistral Vibe, and 5 more coding agents.",
|
||||
"description": "388 production-ready skills across 20 domains (engineering, engineering-core, marketing, product, c-level, c-level-agents, compliance-os, project management, RA/QM, business growth, finance, productivity, marketing top-level, research, research-ops, business-operations, commercial, markdown-html, loop-library, plus standards). 727 Python tools, 842 reference guides, 118 agents (cs-* + personas), 150 slash commands across 99 marketplace plugins. v2.11.2 vendors engineering/skillopt-sleep — a verbatim copy of microsoft/SkillOpt's stdlib-only skillopt_sleep engine + Claude Code plugin surface, giving a local agent a nightly gated self-improvement cycle (read-only session harvest -> mine -> offline replay -> held-out-gated CLAUDE.md/SKILL.md edits -> staged for explicit /skillopt-sleep adopt). productivity/fable-goal (unreleased, post-v2.11.1) converts a rambling description of a desired outcome into one polished /goal prompt for a fresh autonomous session. v2.11.1 turns product-team and project-management into agent-harness domains: fork-orchestrators with deterministic goal routers, a Jira MCP snapshot bridge (Kanban flow metrics + Monte Carlo forecasting), a delegation-governance loop gate, a continuous-discovery cadence tracker, and an Opportunity Solution Tree linter, with /cs:pm and /cs:product command families. v2.10.3 completes the markdown-html domain with md-slides — slide-deck converter (arrow-key / Space / PgDn / Home/End / P keyboard navigation + presenter mode with split-view clock + speaker notes + next-slide preview + URL-hash deep linking like #3 for direct slide jumps + @media print page-per-slide for browser-native PDF export). Reuses md-document's markdown parser; vanilla JS only (no framework runtime); Prism.js opt-in via --syntax. Joins md-review (v2.10.2 code-review converter), md-document (v2.10.1 long-form converter), and the v2.10.0 foundation (orchestrator + design-system). Compatible with Claude Code, Codex CLI, Gemini CLI, Cursor, OpenClaw, Hermes Agent, Mistral Vibe, and 5 more coding agents.",
|
||||
"version": "2.12.0"
|
||||
},
|
||||
"plugins": [
|
||||
|
|
@ -2051,6 +2051,28 @@
|
|||
],
|
||||
"category": "development"
|
||||
},
|
||||
{
|
||||
"name": "spinning-up-deep-rl",
|
||||
"source": "./engineering/spinning-up-deep-rl",
|
||||
"description": "Knowledge base compiled from OpenAI's Spinning Up in Deep RL (MIT, Joshua Achiam) by engineering/book-to-skill. A resident core carries the RL optimization problem, the model-free taxonomy, the policy-gradient template with its five valid weights, the safe-step family (VPG to TRPO to PPO) and the overestimation family (DDPG to TD3 and SAC); 20 on-demand chapters cover key concepts and MDPs, the algorithm taxonomy and model bias, the policy gradient derivation with the log-derivative trick and EGLP lemma, Achiam's researcher essay, the key-papers topic map, the exercises including the silent DDPG broadcasting bug, the benchmark parity disclosure, one chapter per algorithm, and the logger/MPI/ExperimentGrid utilities. Ships a glossary, a patterns file with 16 techniques, a decision cheatsheet with thresholds, a cs-spinning-up-deep-rl agent and /cs:spinning-up-deep-rl. Structured study notes, not a reproduction of the source.",
|
||||
"version": "1.0.0",
|
||||
"author": {
|
||||
"name": "Alireza Rezvani"
|
||||
},
|
||||
"keywords": [
|
||||
"knowledge-base",
|
||||
"book-to-skill",
|
||||
"reinforcement-learning",
|
||||
"deep-rl",
|
||||
"policy-gradient",
|
||||
"ppo",
|
||||
"sac",
|
||||
"td3",
|
||||
"openai-spinning-up",
|
||||
"engineering"
|
||||
],
|
||||
"category": "development"
|
||||
},
|
||||
{
|
||||
"name": "deep-learning-book",
|
||||
"source": "./engineering/deep-learning-book",
|
||||
|
|
|
|||
|
|
@ -1,7 +1,7 @@
|
|||
{
|
||||
"name": "claude-code-skills",
|
||||
"version": "2.12.0",
|
||||
"description": "380 production-ready skills across 20 domains — engineering, marketing, product, compliance, C-level advisory, research, business operations, and more. 706 Python tools, 823 reference guides, 114 agents (cs-* + personas), 138 slash commands, 96 marketplace plugins. The largest open-source skills library for AI coding agents.",
|
||||
"description": "388 production-ready skills across 20 domains — engineering, marketing, product, compliance, C-level advisory, research, business operations, and more. 727 Python tools, 842 reference guides, 118 agents (cs-* + personas), 150 slash commands, 99 marketplace plugins. The largest open-source skills library for AI coding agents.",
|
||||
"author": {
|
||||
"name": "Alireza Rezvani",
|
||||
"url": "https://alirezarezvani.com"
|
||||
|
|
@ -27,8 +27,8 @@
|
|||
"type": "cli",
|
||||
"composerIcon": "./assets/icon.png",
|
||||
"displayName": "Claude Code Skills",
|
||||
"shortDescription": "380 production-ready skills for AI coding agents across 20 domains",
|
||||
"longDescription": "The largest open-source skills library for AI coding agents. 380 skills covering engineering (architecture, DevOps, security, AI/ML, agent tooling), marketing (SEO, AEO, CRO, content), product management, C-level advisory, regulatory compliance (ISO 13485, SOC 2, GDPR), project management, research and research operations, business operations, commercial, finance, and personal productivity. Includes 706 stdlib-only Python CLI tools, 823 reference guides, 114 orchestration agents, and 138 slash commands. Works with Codex, Claude Code, Gemini CLI, Cursor, Hermes Agent, Mistral Vibe, and 7 more tools.",
|
||||
"shortDescription": "388 production-ready skills for AI coding agents across 20 domains",
|
||||
"longDescription": "The largest open-source skills library for AI coding agents. 388 skills covering engineering (architecture, DevOps, security, AI/ML, agent tooling), marketing (SEO, AEO, CRO, content), product management, C-level advisory, regulatory compliance (ISO 13485, SOC 2, GDPR), project management, research and research operations, business operations, commercial, finance, and personal productivity. Includes 727 stdlib-only Python CLI tools, 842 reference guides, 118 orchestration agents, and 150 slash commands. Works with Codex, Claude Code, Gemini CLI, Cursor, Hermes Agent, Mistral Vibe, and 7 more tools.",
|
||||
"developerName": "Alireza Rezvani",
|
||||
"category": "Coding",
|
||||
"capabilities": [
|
||||
|
|
|
|||
43
CHANGELOG.md
43
CHANGELOG.md
|
|
@ -7,6 +7,49 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|||
|
||||
## [Unreleased]
|
||||
|
||||
### Added — engineering/spinning-up-deep-rl: the first book compiled by book-to-skill
|
||||
|
||||
Knowledge-base plugin compiled end-to-end by `engineering/book-to-skill` from OpenAI's
|
||||
[Spinning Up in Deep RL](https://spinningup.openai.com/) (MIT, Copyright (c) 2018 OpenAI;
|
||||
primarily developed by Joshua Achiam). 20 chapters, a glossary, a patterns file and a
|
||||
decision cheatsheet, behind a 2,101-token resident core.
|
||||
|
||||
- **The full pipeline, not a hand-write.** `openai/spinningup` cloned, its `docs/`
|
||||
reStructuredText tree (38 files, ~37k words, ~49K tokens) run through
|
||||
`extract_document.py --mode technical` → analysis → chapter files → supporting files →
|
||||
master `SKILL.md` → `book_skill_validator.py` → `skill_plugin_emitter.py`. The validator
|
||||
passes clean in `--strict` mode and every file is inside budget.
|
||||
- **Rights basis `open-license`, stated and honoured.** The emitter's Step-11 gate refuses a
|
||||
shareable package without one. MIT permits derivative distribution; upstream's notice is
|
||||
reproduced in full in the plugin's `LICENSE` beside this package's own, and `README.md`
|
||||
names the source, the author and the source's frozen version.
|
||||
- **Structure follows the source's own `toctree`.** User documentation (ch01-06), Introduction
|
||||
to RL Parts 1-3 (ch07-09), resources — the researcher essay, key papers, exercises,
|
||||
benchmarks (ch10-13), one chapter per algorithm in lineage order (ch14-19: VPG → TRPO → PPO,
|
||||
DDPG → TD3 → SAC), and the logger / MPI / ExperimentGrid utilities (ch20).
|
||||
- **The cheatsheet carries the judgment a glossary cannot** — the under-5-minute debug
|
||||
turnaround, the 3-seed minimum (10+ to be thorough), family-specific benchmark network
|
||||
defaults, and Spinning Up's own parity disclosure: DDPG/TD3/SAC are research-grade,
|
||||
VPG/TRPO/PPO are not, and the docs say to use OpenAI Baselines for those.
|
||||
- **Counters:** skills 387 → 388; agents 117 → 118; commands 149 → 150; plugins 98 → 99.
|
||||
Tools and references unchanged by this plugin — a compiled knowledge base ships notes, not
|
||||
scripts. (These sit on top of `deep-learning-book`, which merged into `dev` first; the
|
||||
derived totals are 388 skills / 727 tools / 842 references / 118 agents / 150 commands /
|
||||
99 plugins.)
|
||||
|
||||
### Fixed — book-to-skill's plugin emitter produced manifests this repo's CI rejects
|
||||
|
||||
`skill_plugin_emitter.py` wrote its whole `source` provenance block into `plugin.json`, with an
|
||||
inline comment asserting that `source` and `attribution` were approved extension fields. That had
|
||||
been true and no longer was: Claude Code rejects an entire manifest on any unrecognized key
|
||||
(issue #954), and `scripts/check_plugin_json.py` hard-fails such a manifest, pointing at
|
||||
`.claude-plugin/authoring-notes.json` instead. Every package the emitter produced therefore failed
|
||||
the blocking CI gate the moment it was committed — a defect at the very last step of the pipeline,
|
||||
which is why it had gone unnoticed. `_plugin_manifest()` now emits spec fields only and a new
|
||||
`_authoring_notes()` writes the sidecar. Recorded as deviation 26 in
|
||||
`engineering/book-to-skill/README.md`. The printed `marketplace.json` snippet is unchanged: `source`
|
||||
is a valid key there, which is how it leaked into the manifest originally.
|
||||
|
||||
### Added — engineering/deep-learning-book: a companion to the free Deep Learning textbook
|
||||
|
||||
New `engineering/deep-learning-book/` plugin: a study companion for *Deep Learning* by
|
||||
|
|
|
|||
51
CLAUDE.md
51
CLAUDE.md
File diff suppressed because one or more lines are too long
12
README.md
12
README.md
|
|
@ -1,6 +1,6 @@
|
|||
# Claude Code Skills & Plugins — Agent Skills for Every Coding Tool
|
||||
|
||||
**387 production-ready Claude Code skills, plugins, and agent skills for 13 AI coding tools.**
|
||||
**388 production-ready Claude Code skills, plugins, and agent skills for 13 AI coding tools.**
|
||||
|
||||
The most comprehensive open-source library of Claude Code skills and agent plugins — also works with OpenAI Codex, Gemini CLI, Cursor, and 9 more coding agents. Reusable expertise packages covering engineering, DevOps, marketing (incl. AEO — Answer Engine Optimization for LLM citation), security (PreToolUse hooks), compliance, C-level advisory (incl. founder-mode CFO/CMO/CRO/CPO/COO/CHRO/CISO/GC/CDO/CAIO/CCO/VPE personas + 21 /cs:* slash commands), productivity (capture/email/reflect/weekly-review/deep-work/meetings), an academic research stack (litreview/grants/dossier/patent/syllabus/pulse/notebooklm/deep-research + hybrid router), and enterprise Research Operations (clinical-research/research-finance/market-research/product-research, v2.9.0).
|
||||
|
||||
|
|
@ -10,10 +10,10 @@ The most comprehensive open-source library of Claude Code skills and agent plugi
|
|||
[^vibe]: Mistral Vibe is also **BYO-sync tier**: the repo ships a pre-generated `.vibe/skills/claude-skills/` tree, run `./scripts/vibe-install.sh` once locally to install into `~/.vibe/skills/`. Same agentskills.io SKILL.md standard — no format conversion. Docs: <https://docs.mistral.ai/mistral-vibe/agents-skills>.
|
||||
|
||||
[](https://opensource.org/licenses/MIT)
|
||||
[](#skills-overview)
|
||||
[](#agents)
|
||||
[](#skills-overview)
|
||||
[](#agents)
|
||||
[](#personas)
|
||||
[](#commands)
|
||||
[](#commands)
|
||||
[](https://github.com/alirezarezvani/claude-skills/stargazers)
|
||||
[](https://getskillcheck.com)
|
||||
|
||||
|
|
@ -152,12 +152,12 @@ Run `./scripts/convert.sh --tool all` to generate tool-specific outputs locally.
|
|||
|
||||
## Skills Overview
|
||||
|
||||
**387 production-ready skills across 20 domains:**
|
||||
**388 production-ready skills across 20 domains:**
|
||||
|
||||
| Domain | Skills | Highlights | Details |
|
||||
|--------|--------|------------|---------|
|
||||
| **🔧 Engineering — Core** | 53 | Architecture, frontend, backend, fullstack, QA, DevOps, SecOps, AI/ML, data, Playwright Pro (test gen, flaky fix, migrations), self-improving agent (auto-memory curation), security suite, a11y audit, **named-persona-adversarial-review** (review via named engineering philosophies), **embedded-iot-mentor** (MCU/board selection, firmware-reuse-first, breadboard-MVP discipline) | [engineering-team/](engineering-team/) |
|
||||
| **⚡ Engineering — POWERFUL** | 92 | Agent designer, RAG architect, database designer, CI/CD builder, security auditor, MCP builder, AgentHub, Helm charts, Terraform, self-eval, llm-wiki, tc-tracker, autoresearch-agent, **reliability portfolio** (feature-flags-architect, kubernetes-operator, chaos-engineering, slo-architect), ship-gate, security-guidance PreToolUse hook, **Matt Pocock skills** (write-a-skill, caveman, grill-me, handoff, grill-with-docs), **zero-hallucination-coder** (Discuss→Map→Decompose→Execute→Verify), **agent-harness** (goal→plan→execute→verify→close loops over any domain), **memory-engineering** (price the memory write path, pick which cost to pay, audit FACT/SKILL/LOG density, gate on a forgetting policy), **skillopt-sleep** (nightly gated self-evolution from real Claude Code sessions, vendored from microsoft/SkillOpt), **book-to-skill** (compile a book, docs folder, or spec collection into a knowledge-base skill, then package it as a plugin), **boost-asio-pro** (async C++ networking — version-gated coroutine/callback styles, strand discipline), **human-gate** (batched human review as a structured artifact + a gate that refuses to close on open blockers), **agent-memory** (four-tier L0-L3 memory ladder over Claude Code hooks; promotion earned by recurrence across sessions and days, redacted or contested claims refuse to promote, nothing reaches CLAUDE.md without a human adopt), **skill-doctor** (grade the agent setup from real session history — rubric-judged, deterministically gated, evidence-traced skill edits), **deep-learning-book** (companion to the free Goodfellow/Bengio/Courville textbook — 20 chapter files with a 2016→2026 delta layer, reading-path planner, Ch-11 training diagnostic, capacity planner, FLOP/activation-memory calculator) | [engineering/](engineering/) |
|
||||
| **⚡ Engineering — POWERFUL** | 93 | Agent designer, RAG architect, database designer, CI/CD builder, security auditor, MCP builder, AgentHub, Helm charts, Terraform, self-eval, llm-wiki, tc-tracker, autoresearch-agent, **reliability portfolio** (feature-flags-architect, kubernetes-operator, chaos-engineering, slo-architect), ship-gate, security-guidance PreToolUse hook, **Matt Pocock skills** (write-a-skill, caveman, grill-me, handoff, grill-with-docs), **zero-hallucination-coder** (Discuss→Map→Decompose→Execute→Verify), **agent-harness** (goal→plan→execute→verify→close loops over any domain), **memory-engineering** (price the memory write path, pick which cost to pay, audit FACT/SKILL/LOG density, gate on a forgetting policy), **skillopt-sleep** (nightly gated self-evolution from real Claude Code sessions, vendored from microsoft/SkillOpt), **book-to-skill** (compile a book, docs folder, or spec collection into a knowledge-base skill, then package it as a plugin), **boost-asio-pro** (async C++ networking — version-gated coroutine/callback styles, strand discipline), **human-gate** (batched human review as a structured artifact + a gate that refuses to close on open blockers), **agent-memory** (four-tier L0-L3 memory ladder over Claude Code hooks; promotion earned by recurrence across sessions and days, redacted or contested claims refuse to promote, nothing reaches CLAUDE.md without a human adopt), **skill-doctor** (grade the agent setup from real session history — rubric-judged, deterministically gated, evidence-traced skill edits), **deep-learning-book** (companion to the free Goodfellow/Bengio/Courville textbook — 20 chapter files with a 2016→2026 delta layer, reading-path planner, Ch-11 training diagnostic, capacity planner, FLOP/activation-memory calculator), **spinning-up-deep-rl** (knowledge base compiled from OpenAI's Spinning Up in Deep RL — RL fundamentals, the model-free taxonomy, policy-gradient derivations, and one chapter per reference algorithm) | [engineering/](engineering/) |
|
||||
| **🎯 Product** | 17 | Product manager, agile PO, strategist, UX researcher, UI design, landing pages, SaaS scaffolder, analytics, experiment designer, discovery, roadmap communicator, code-to-prd, apple-hig-expert | [product-team/](product-team/) |
|
||||
| **📣 Marketing** | 49 | 8 pods: Content, SEO + AEO (`aeo` — E-E-A-T audit, citation tracking across 5 LLMs) + local (`local-seo-manager` — GBP/NAP/Map-Pack), CRO, Channels, Growth, Intelligence, Sales + `business-name-fit` (cross-cultural naming) + context foundation + orchestration router | [marketing-skill/](marketing-skill/) |
|
||||
| **🚀 Productivity** | 12 | `capture` (brain-dump-to-action), `email` pair (inbox-setup + inbox-triage), `reflect` (journal), `handoff` (Matt Pocock-inspired), `andreessen` (market-first decision mode), `roast` (5-angle idea panel → GO/RESHAPE/KILL), `fable-goal` (ramble → autonomous /goal prompt), `weekly-review` (GTD loop with refusal gate), `deep-work` (time-blocking + shallow-work budget), `meetings` (cost gate + agenda + action items), `swedish-mentor` (CEFR-leveled Swedish learning paths) | [productivity/](productivity/) |
|
||||
|
|
|
|||
|
|
@ -3,6 +3,6 @@
|
|||
"derived_from": "https://github.com/virgiliojr94/book-to-skill",
|
||||
"original_author": "virgiliojr94",
|
||||
"original_license": "MIT",
|
||||
"derivation_note": "The extraction library (book_to_skill/ — config, exceptions, sanitize, dependencies, utils, and the 7 per-format parsers) is vendored from upstream and carries upstream's multi-format chains, chapter detection across 5 script families, and Unicode/XXE hardening. 25 numbered deviations are recorded in README.md, which is the authoritative list: the workflow was rewritten Claude-Code-first for this repo's conventions, the 3 upstream tools were reworked into 4 argparse CLIs with --help/--sample, install-on-import behaviour was replaced with a report-only default, and Step 11 (plugin emission with a rights gate) plus the validator's budget and index families are additions with no upstream counterpart."
|
||||
"derivation_note": "The extraction library (book_to_skill/ — config, exceptions, sanitize, dependencies, utils, and the 7 per-format parsers) is vendored from upstream and carries upstream's multi-format chains, chapter detection across 5 script families, and Unicode/XXE hardening. 26 numbered deviations are recorded in README.md, which is the authoritative list: the workflow was rewritten Claude-Code-first for this repo's conventions, the 3 upstream tools were reworked into 4 argparse CLIs with --help/--sample, install-on-import behaviour was replaced with a report-only default, and Step 11 (plugin emission with a rights gate) plus the validator's budget and index families are additions with no upstream counterpart."
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -101,7 +101,7 @@ small; just read it."
|
|||
|
||||
## Deviations from upstream
|
||||
|
||||
**This numbered list is the authoritative record.** `plugin.json`'s
|
||||
**This numbered list is the authoritative record.** `.claude-plugin/authoring-notes.json`'s
|
||||
`attribution.derivation_note` summarizes it; if the two ever disagree, this list wins.
|
||||
|
||||
**Structural**
|
||||
|
|
@ -245,9 +245,10 @@ small; just read it."
|
|||
preventing hand-edit mistakes contradicted the manifest beside it); the symlink guard is
|
||||
backed by a **post-copy re-walk** that deletes the package if a link appeared during the
|
||||
copy, closing the check-then-act window rather than only narrowing it; and the manifest
|
||||
carries `source.license_scope` stating that the top-level `license` covers the package
|
||||
emits `source.license_scope` stating that the top-level `license` covers the package
|
||||
scaffolding, not the compiled notes — a distinction that previously lived only in README
|
||||
prose where a tool reading the manifest alone would miss it.
|
||||
prose where a tool reading the metadata alone would miss it. (Deviation 26 later moved
|
||||
that whole `source` block out of `plugin.json` and into the sidecar.)
|
||||
|
||||
21. **The documented quick-start actually runs.** SKILL.md's copy-paste block referenced
|
||||
`$WORKDIR` and `$SKILLS_HOME` without ever defining them — following it literally produced
|
||||
|
|
@ -297,6 +298,21 @@ small; just read it."
|
|||
leaves the victim's contents intact and is replaced by a 0600 file we own. Degrades to the
|
||||
previous path-based checks on platforms without `dir_fd`/`O_NOFOLLOW` (Windows).
|
||||
|
||||
26. **Provenance moved out of `plugin.json` into the sidecar the repo actually allows.**
|
||||
The manifest builder wrote its whole `source` block (spec, build pattern, source document,
|
||||
chapter count, distribution, `license_scope`, `rights_basis`) into `plugin.json`, and an
|
||||
inline comment asserted that `source` and `attribution` were "approved extension fields."
|
||||
That had been true and no longer was: Claude Code rejects an entire manifest on any
|
||||
unrecognized key (issue #954), and this repo's own `scripts/check_plugin_json.py` hard-fails
|
||||
a `plugin.json` carrying either field, pointing at `.claude-plugin/authoring-notes.json`
|
||||
instead. So every package the emitter produced failed the repo's blocking CI gate the moment
|
||||
it was committed — a defect that only surfaces at the very last step of the pipeline, which
|
||||
is why it survived. `_plugin_manifest()` now emits spec fields only, and a new
|
||||
`_authoring_notes()` writes the `source` block to `.claude-plugin/authoring-notes.json`.
|
||||
Note that `source` remains a *valid* key in a `marketplace.json` `plugins[]` entry, which is
|
||||
how it leaked into the manifest in the first place; the printed marketplace snippet is
|
||||
unchanged and still correct.
|
||||
|
||||
---
|
||||
|
||||
## Security audit
|
||||
|
|
|
|||
|
|
@ -382,10 +382,23 @@ Emits `<domain>/<slug>/` with `.claude-plugin/plugin.json`, `README.md`,
|
|||
`skills/<slug>/`, then prints the marketplace entry to register. Drop `--dry-run` to write.
|
||||
|
||||
**Rights gate.** The emitter defaults to `--distribution local`, which records
|
||||
`source.cleared_for_distribution: false` in the manifest. `--distribution shareable` **refuses** unless `--rights` names a basis:
|
||||
`source.cleared_for_distribution: false` in `.claude-plugin/authoring-notes.json`.
|
||||
`--distribution shareable` **refuses** unless `--rights` names a basis:
|
||||
`public-domain`, `open-license`, `internal-docs`, or `author-permission`. Fair use is
|
||||
deliberately not an option — it is a defence, not a licence, and not this tool's call.
|
||||
|
||||
**Attribution is yours to add, by hand, when the source is someone else's work.** The emitter
|
||||
writes a `source` block — how the skill was built — because that is all it can know; it has
|
||||
`--source-note` free text and a rights basis, not an upstream URL, author or licence, and a
|
||||
half-filled `attribution` block is worse than none. So when `--rights` is anything but
|
||||
`internal-docs`, add an `attribution` block beside it in `authoring-notes.json`
|
||||
(`derived_from`, `original_author`, `original_license`, `original_copyright`,
|
||||
`derivation_note` — the shape the rest of this repo uses), and put the upstream licence
|
||||
notice in the package's `LICENSE` and a credit line in its `README.md`. **That last part is
|
||||
the obligation:** `authoring-notes.json` is authoring metadata Claude Code never reads, and a
|
||||
sidecar JSON file is not a licence notice. `engineering/spinning-up-deep-rl` is the worked
|
||||
example.
|
||||
|
||||
The emitter also refuses to wrap a skill with validation errors. Fix the source skill first.
|
||||
|
||||
Registration in `.claude-plugin/marketplace.json` stays manual — it is a repo-wide change.
|
||||
|
|
|
|||
|
|
@ -10,7 +10,8 @@ This tool closes that gap. Point it at a generated book skill and it emits the
|
|||
plugin package this repo's conventions require:
|
||||
|
||||
<domain>/<slug>/
|
||||
├── .claude-plugin/plugin.json manifest with `./skills/<slug>` + attribution
|
||||
├── .claude-plugin/plugin.json manifest, spec fields only (no `source`)
|
||||
├── .claude-plugin/authoring-notes.json provenance the manifest may not carry
|
||||
├── README.md what the skill knows and where it came from
|
||||
├── agents/cs-<slug>.md persona that answers from the book
|
||||
├── commands/cs-<slug>.md /cs:<slug> entry point
|
||||
|
|
@ -24,7 +25,8 @@ Rights gate: a book skill built from a copyrighted work is personal study
|
|||
notes. `--distribution shareable` therefore refuses to emit unless `--rights`
|
||||
names a basis that permits redistribution. `--distribution local` (the default)
|
||||
emits with a notice and records `source.cleared_for_distribution: false`
|
||||
in the manifest.
|
||||
in `authoring-notes.json`. Provenance never goes in `plugin.json`: Claude Code
|
||||
rejects the whole manifest on any unrecognized key (issue #954).
|
||||
|
||||
Exit codes:
|
||||
0 package emitted (or --dry-run / --sample completed)
|
||||
|
|
@ -166,13 +168,13 @@ def _assert_no_symlinks(skill_dir: Path) -> None:
|
|||
|
||||
|
||||
def _plugin_manifest(identity: dict, *, domain: str, author: str, author_url: str,
|
||||
repository: str, distribution: str, rights: str | None,
|
||||
source_note: str) -> dict:
|
||||
# Key set and the two approved extension fields (`source`, `attribution`) are
|
||||
# fixed by scripts/check_plugin_json.py. Anything else — including a
|
||||
# top-level "private" flag — fails the repo's CI gate, so the distribution
|
||||
# marker lives inside `source`.
|
||||
manifest = {
|
||||
repository: str) -> dict:
|
||||
# Spec fields ONLY. Claude Code's manifest validator rejects the whole
|
||||
# plugin.json on any unrecognized key (issue #954), and the repo's own gate
|
||||
# (scripts/check_plugin_json.py) hard-fails a manifest carrying `source` or
|
||||
# `attribution`. Provenance therefore goes to the sidecar file the validator
|
||||
# never reads — see _authoring_notes below.
|
||||
return {
|
||||
"name": identity["slug"],
|
||||
"description": identity["description"],
|
||||
"version": "1.0.0",
|
||||
|
|
@ -185,31 +187,42 @@ def _plugin_manifest(identity: dict, *, domain: str, author: str, author_url: st
|
|||
# the manifest sees the distinction that README.md makes in prose.
|
||||
"license": "MIT",
|
||||
"skills": [f"./skills/{identity['slug']}"],
|
||||
"source": {
|
||||
"spec": "generated by engineering/book-to-skill",
|
||||
"build_pattern": "book-to-skill conversion: extract -> analyze -> chapter files "
|
||||
"+ glossary + patterns + cheatsheet -> master SKILL.md",
|
||||
"distinct_from": "not a hand-authored skill; every claim traces to the source "
|
||||
"document named below",
|
||||
"source_document": source_note or identity["title"],
|
||||
"chapters": identity["chapters"],
|
||||
"distribution": distribution,
|
||||
"license_scope": (
|
||||
"The top-level `license` covers this package's scaffolding only. The "
|
||||
"compiled notes under skills/ are derived from the source document and "
|
||||
"carry that work's terms; see source.rights_basis."
|
||||
),
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def _authoring_notes(identity: dict, *, distribution: str, rights: str | None,
|
||||
source_note: str) -> dict:
|
||||
"""Provenance for `.claude-plugin/authoring-notes.json`.
|
||||
|
||||
The repo's schema allows exactly two keys here, `source` and `attribution`;
|
||||
a generated skill's provenance is a `source` block. This file is authoring
|
||||
metadata that Claude Code's manifest validator never reads, which is
|
||||
precisely why the fields live here rather than in plugin.json.
|
||||
"""
|
||||
source = {
|
||||
"spec": "generated by engineering/book-to-skill",
|
||||
"build_pattern": "book-to-skill conversion: extract -> analyze -> chapter files "
|
||||
"+ glossary + patterns + cheatsheet -> master SKILL.md",
|
||||
"distinct_from": "not a hand-authored skill; every claim traces to the source "
|
||||
"document named below",
|
||||
"source_document": source_note or identity["title"],
|
||||
"chapters": identity["chapters"],
|
||||
"distribution": distribution,
|
||||
"license_scope": (
|
||||
"plugin.json's top-level `license` covers this package's scaffolding only. "
|
||||
"The compiled notes under skills/ are derived from the source document and "
|
||||
"carry that work's terms; see source.rights_basis."
|
||||
),
|
||||
}
|
||||
if rights:
|
||||
manifest["source"]["rights_basis"] = rights
|
||||
manifest["source"]["rights_note"] = RIGHTS_BASES[rights]
|
||||
source["rights_basis"] = rights
|
||||
source["rights_note"] = RIGHTS_BASES[rights]
|
||||
if distribution == "local":
|
||||
# A local package is study notes from a work the user owns. The flag is
|
||||
# advisory metadata, not enforcement — it exists so a later publish step
|
||||
# (or a reviewer) can see the package was never cleared for sharing.
|
||||
manifest["source"]["cleared_for_distribution"] = False
|
||||
return manifest
|
||||
source["cleared_for_distribution"] = False
|
||||
return {"source": source}
|
||||
|
||||
|
||||
def _agent_markdown(identity: dict, domain: str, source_note: str) -> str:
|
||||
|
|
@ -407,10 +420,12 @@ def emit(*, skill_dir: Path, dest_root: Path, domain: str, author: str, author_u
|
|||
_assert_replaceable(package_root, resolved_dest, skill_dir)
|
||||
|
||||
manifest = _plugin_manifest(identity, domain=domain, author=author, author_url=author_url,
|
||||
repository=repository, distribution=distribution, rights=rights,
|
||||
source_note=source_note)
|
||||
repository=repository)
|
||||
notes = _authoring_notes(identity, distribution=distribution, rights=rights,
|
||||
source_note=source_note)
|
||||
files = {
|
||||
".claude-plugin/plugin.json": json.dumps(manifest, indent=2, ensure_ascii=False) + "\n",
|
||||
".claude-plugin/authoring-notes.json": json.dumps(notes, indent=2, ensure_ascii=False) + "\n",
|
||||
"README.md": _readme_markdown(identity, domain=domain, distribution=distribution,
|
||||
rights=rights, source_note=source_note),
|
||||
f"agents/cs-{identity['slug']}.md": _agent_markdown(identity, domain, source_note),
|
||||
|
|
|
|||
|
|
@ -0,0 +1,22 @@
|
|||
{
|
||||
"source": {
|
||||
"spec": "generated by engineering/book-to-skill",
|
||||
"build_pattern": "book-to-skill conversion: extract -> analyze -> chapter files + glossary + patterns + cheatsheet -> master SKILL.md",
|
||||
"distinct_from": "not a hand-authored skill; every claim traces to the source document named below",
|
||||
"source_document": "Spinning Up in Deep RL by Joshua Achiam (OpenAI)",
|
||||
"chapters": 20,
|
||||
"distribution": "shareable",
|
||||
"license_scope": "plugin.json's top-level `license` covers this package's scaffolding only. The compiled notes under skills/ are derived from the source document and carry that work's terms; see source.rights_basis.",
|
||||
"rights_basis": "open-license",
|
||||
"rights_note": "the source work carries a licence permitting derivative distribution"
|
||||
},
|
||||
"attribution": {
|
||||
"derived_from": "https://github.com/openai/spinningup",
|
||||
"upstream_docs": "https://spinningup.openai.com/",
|
||||
"upstream_path": "docs/ (reStructuredText tree, 38 files)",
|
||||
"original_author": "Joshua Achiam, OpenAI",
|
||||
"original_license": "MIT",
|
||||
"original_copyright": "Copyright (c) 2018 OpenAI (http://openai.com)",
|
||||
"derivation_note": "No upstream code or prose is reproduced. The compiled skill is structured study notes -- named frameworks, key concepts, decision rules and per-chapter summaries -- generated by engineering/book-to-skill from the source's docs/ tree, with chapter structure taken from the source's own toctree. Upstream's MIT notice is reproduced in full in this package's LICENSE, which is where the licence obligation is met; this file is authoring metadata and is not a licence notice. Coverage is pinned to the source's final (January 2020 PyTorch) update."
|
||||
}
|
||||
}
|
||||
15
engineering/spinning-up-deep-rl/.claude-plugin/plugin.json
Normal file
15
engineering/spinning-up-deep-rl/.claude-plugin/plugin.json
Normal file
|
|
@ -0,0 +1,15 @@
|
|||
{
|
||||
"name": "spinning-up-deep-rl",
|
||||
"description": "Knowledge base from \"Spinning Up in Deep RL\" by Joshua Achiam (OpenAI, MIT-licensed). Use when applying Achiam's frameworks for RL fundamentals and MDPs, the model-free algorithm taxonomy, policy gradient derivations, the six reference algorithms (VPG, TRPO, PPO, DDPG, TD3, SAC), debugging silently-failing RL code, or running rigorous multi-seed RL experiments.",
|
||||
"version": "1.0.0",
|
||||
"author": {
|
||||
"name": "Alireza Rezvani",
|
||||
"url": "https://github.com/alirezarezvani"
|
||||
},
|
||||
"homepage": "https://github.com/alirezarezvani/claude-skills/tree/main/engineering/spinning-up-deep-rl",
|
||||
"repository": "https://github.com/alirezarezvani/claude-skills",
|
||||
"license": "MIT",
|
||||
"skills": [
|
||||
"./skills/spinning-up-deep-rl"
|
||||
]
|
||||
}
|
||||
58
engineering/spinning-up-deep-rl/LICENSE
Normal file
58
engineering/spinning-up-deep-rl/LICENSE
Normal file
|
|
@ -0,0 +1,58 @@
|
|||
This package has two layers with two different terms.
|
||||
|
||||
------------------------------------------------------------------------------
|
||||
1. Package scaffolding — plugin manifest, agent, command, README
|
||||
------------------------------------------------------------------------------
|
||||
|
||||
MIT License
|
||||
|
||||
Copyright (c) 2026 Alireza Rezvani
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the "Software"), to deal
|
||||
in the Software without restriction, including without limitation the rights
|
||||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||
copies of the Software, and to permit persons to whom the Software is
|
||||
furnished to do so, subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in
|
||||
all copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
|
||||
THE SOFTWARE.
|
||||
|
||||
------------------------------------------------------------------------------
|
||||
2. Compiled notes under skills/spinning-up-deep-rl/
|
||||
------------------------------------------------------------------------------
|
||||
|
||||
Derived from OpenAI's "Spinning Up in Deep RL" documentation
|
||||
(https://spinningup.openai.com/ · https://github.com/openai/spinningup),
|
||||
primarily developed by Joshua Achiam. Distributed under the same MIT terms,
|
||||
whose notice is reproduced below in full as that licence requires.
|
||||
|
||||
The MIT License
|
||||
|
||||
Copyright (c) 2018 OpenAI (http://openai.com)
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the "Software"), to deal
|
||||
in the Software without restriction, including without limitation the rights
|
||||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||
copies of the Software, and to permit persons to whom the Software is
|
||||
furnished to do so, subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in
|
||||
all copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
|
||||
THE SOFTWARE.
|
||||
49
engineering/spinning-up-deep-rl/README.md
Normal file
49
engineering/spinning-up-deep-rl/README.md
Normal file
|
|
@ -0,0 +1,49 @@
|
|||
# Spinning Up in Deep RL
|
||||
|
||||
Knowledge-base plugin compiled from **Spinning Up in Deep RL by Joshua Achiam (OpenAI)** by
|
||||
[`engineering/book-to-skill`](../book-to-skill/). 20 chapters indexed.
|
||||
|
||||
## What is in here
|
||||
|
||||
| File | Contents |
|
||||
|------|----------|
|
||||
| `skills/spinning-up-deep-rl/SKILL.md` | Core frameworks, chapter index, topic index (resident, under 4k tokens) |
|
||||
| `skills/spinning-up-deep-rl/chapters/` | One summary per chapter — loaded on demand, never all at once |
|
||||
| `skills/spinning-up-deep-rl/glossary.md` | Every significant term, alphabetized, with its chapter |
|
||||
| `skills/spinning-up-deep-rl/patterns.md` | Techniques and design patterns with trade-offs |
|
||||
| `skills/spinning-up-deep-rl/cheatsheet.md` | Decision rules, thresholds and trade-off matrices |
|
||||
|
||||
## Use
|
||||
|
||||
```
|
||||
/cs:spinning-up-deep-rl # core frameworks + chapter index
|
||||
/cs:spinning-up-deep-rl <topic> # resolve via topic index, read one chapter
|
||||
/cs:spinning-up-deep-rl ch05 # read one chapter summary
|
||||
```
|
||||
|
||||
Or invoke the `cs-spinning-up-deep-rl` agent for a working session anchored to this source.
|
||||
|
||||
## Provenance and limits
|
||||
|
||||
**Source:** OpenAI's [Spinning Up in Deep RL](https://spinningup.openai.com/)
|
||||
([openai/spinningup](https://github.com/openai/spinningup)), primarily developed by
|
||||
**Joshua Achiam**. Compiled from the `docs/` reStructuredText tree at the January 2020
|
||||
PyTorch update.
|
||||
|
||||
**Rights basis:** `open-license`. The source is **MIT, Copyright (c) 2018 OpenAI**, which
|
||||
permits derivative distribution. The full upstream notice is reproduced in
|
||||
[`LICENSE`](LICENSE) alongside this package's own; the top-level `license` field in
|
||||
`plugin.json` covers the scaffolding only.
|
||||
|
||||
Generated, not hand-authored: every claim traces to the source document. It carries that source's
|
||||
blind spots, and it is a set of structured notes — **not a copy of the work and not a substitute
|
||||
for reading it**.
|
||||
|
||||
**What it does not cover:** DQN and the discrete-action value-learning family, recurrent or
|
||||
convolutional architectures, partially-observed settings, model-based implementations, and any
|
||||
deep RL work after early 2020. The six implementations documented are educational; `ch13` records
|
||||
which are research-grade (DDPG, TD3, SAC) and which are not (VPG, TRPO, PPO).
|
||||
|
||||
Distribution: `shareable`. Regenerate or extend with
|
||||
`python3 engineering/book-to-skill/skills/book-to-skill/scripts/extract_document.py`, then re-run
|
||||
`book_skill_validator.py` before loading the result.
|
||||
|
|
@ -0,0 +1,37 @@
|
|||
---
|
||||
name: cs-spinning-up-deep-rl
|
||||
description: Answers from the knowledge base compiled from Spinning Up in Deep RL by Joshua Achiam (OpenAI). Loads the master frameworks first and reads a single chapter file on demand rather than the whole source. Refuses to answer beyond what the source covers.
|
||||
skills: engineering/spinning-up-deep-rl/skills/spinning-up-deep-rl
|
||||
domain: engineering
|
||||
model: opus
|
||||
tools: [Read, Grep, Glob]
|
||||
---
|
||||
|
||||
# Spinning Up in Deep RL — Knowledge Agent
|
||||
|
||||
## Voice
|
||||
|
||||
**Opening:** "Which framework or chapter are you reaching for?"
|
||||
**Forcing question:** "Is this something the source actually covers, or are you asking me to
|
||||
extrapolate past it?"
|
||||
**Closing:** "That is the author's formulation, from ch<N>. Anything past it is my inference, not theirs."
|
||||
|
||||
## Purpose
|
||||
|
||||
Applies the frameworks compiled from **Spinning Up in Deep RL by Joshua Achiam (OpenAI)** (20 chapters
|
||||
indexed) while the user works. Answers with the author's exact naming, then cites the chapter.
|
||||
|
||||
## How it navigates
|
||||
|
||||
1. Read `skills/spinning-up-deep-rl/SKILL.md` — Core Frameworks and both indexes.
|
||||
2. Match the question against the Topic Index; read **only** the chapter files it points to.
|
||||
3. Reach for `glossary.md` for a term, `patterns.md` for a technique, `cheatsheet.md` for a decision.
|
||||
4. Never load every chapter — that is the cost this skill exists to avoid.
|
||||
|
||||
## Hard rules
|
||||
|
||||
- **Cite the chapter.** Every framework claim names the chapter it came from.
|
||||
- **Do not extrapolate silently.** If the source does not cover it, say so before answering from
|
||||
general knowledge, and label which is which.
|
||||
- **Preserve exact naming.** The author's term is the interface; a paraphrase breaks lookup.
|
||||
- **Do not reproduce the source at length.** These are structured notes, not a copy of the work.
|
||||
|
|
@ -0,0 +1,30 @@
|
|||
---
|
||||
name: "cs-spinning-up-deep-rl"
|
||||
description: "/cs:spinning-up-deep-rl [topic | framework | chNN] — query the knowledge base compiled from Spinning Up in Deep RL by Joshua Achiam (OpenAI). Use when applying its frameworks while working, looking up a term, or reading one chapter's summary."
|
||||
---
|
||||
|
||||
# /cs:spinning-up-deep-rl — Spinning Up in Deep RL
|
||||
|
||||
**Command:** `/cs:spinning-up-deep-rl [topic | framework name | chNN]`
|
||||
|
||||
## When to run
|
||||
|
||||
- Applying a framework from this source to work in progress
|
||||
- Looking up the author's exact formulation of a term
|
||||
- Reading one chapter's compiled summary without opening the source
|
||||
- Checking whether the source covers a question at all
|
||||
|
||||
## What it does
|
||||
|
||||
1. Loads `engineering/spinning-up-deep-rl/skills/spinning-up-deep-rl/SKILL.md` — Core Frameworks plus the Chapter and Topic indexes.
|
||||
2. **No argument** → reports the core frameworks and the chapter index.
|
||||
3. **A topic or framework name** → resolves it through the Topic Index and reads only the
|
||||
matching chapter file.
|
||||
4. **`chNN`** → reads that chapter's summary directly.
|
||||
5. Answers with the author's naming and cites the chapter.
|
||||
|
||||
## Boundary
|
||||
|
||||
This command answers from **one source** (20 chapters indexed). Anything it does not
|
||||
cover gets said out loud rather than filled in — and hands-on work in your codebase belongs to the
|
||||
engineering skills, not here.
|
||||
|
|
@ -0,0 +1,192 @@
|
|||
---
|
||||
name: spinning-up-deep-rl
|
||||
description: "Knowledge base from \"Spinning Up in Deep RL\" by Joshua Achiam (OpenAI, MIT-licensed). Use when applying Achiam's frameworks for RL fundamentals and MDPs, the model-free algorithm taxonomy, policy gradient derivations, the six reference algorithms (VPG, TRPO, PPO, DDPG, TD3, SAC), debugging silently-failing RL code, or running rigorous multi-seed RL experiments."
|
||||
---
|
||||
|
||||
# Spinning Up in Deep RL
|
||||
|
||||
**Author**: Joshua Achiam (OpenAI) | **Source**: spinningup.readthedocs.io, MIT | **Chapters**: 20 | **Generated**: 2026-08-25
|
||||
|
||||
## How to Use This Skill
|
||||
|
||||
- **No argument** — load the core frameworks below
|
||||
- **A topic** — ask about `advantage function`, `target networks`, `entropy regularization`;
|
||||
I resolve it through the Topic Index and read that chapter file
|
||||
- **`chNN`** — I load that chapter's summary
|
||||
- **"what chapters do you have?"** — the full index
|
||||
|
||||
```
|
||||
/cs:spinning-up-deep-rl # core frameworks + chapter index
|
||||
/cs:spinning-up-deep-rl entropy regularization # topic index -> ch19, read that chapter
|
||||
/cs:spinning-up-deep-rl ch09 # one chapter summary
|
||||
```
|
||||
|
||||
When you ask about something not in Core Frameworks, I read the relevant chapter file before
|
||||
answering rather than guessing from the index.
|
||||
|
||||
---
|
||||
|
||||
## Core Frameworks & Mental Models
|
||||
|
||||
### The RL problem
|
||||
`pi* = argmax_pi J(pi)`, where `J(pi) = E_{tau~pi}[R(tau)]`. Every algorithm approximates
|
||||
this; where it substitutes a different objective (a Bellman residual, a surrogate), that
|
||||
substitution is the source of its failure modes. Four value functions — `V^pi`, `Q^pi`, `V*`,
|
||||
`Q*` — all obey Bellman self-consistency, and `a*(s) = argmax_a Q*(s,a)` is why Q-learning
|
||||
is a viable family at all. Advantage `A^pi(s,a) = Q^pi(s,a) - V^pi(s)` is the relative-quality
|
||||
signal policy gradients run on. (ch07)
|
||||
|
||||
### The two branching questions
|
||||
Place any algorithm by asking: **does it have or learn a model**, and **what does it learn**
|
||||
(policy, Q-function, value function, model). That generates the whole landscape. (ch08)
|
||||
|
||||
### Policy optimization vs Q-learning — the central trade-off
|
||||
- **Policy optimization** is *principled: you directly optimize the thing you want.* Stable
|
||||
and reliable. On-policy, so it cannot reuse data, so it is sample-hungry.
|
||||
- **Q-learning** only *indirectly* optimizes performance, by training `Q_theta` to satisfy a
|
||||
self-consistency equation. Many failure modes, so less stable. But substantially more
|
||||
sample efficient when it works, because it reuses everything.
|
||||
- **Satisfying the Bellman equations well carries no guarantee of good policy performance.**
|
||||
- The two are not exclusive — DDPG and SAC live between them deliberately. (ch08)
|
||||
|
||||
### The policy gradient template
|
||||
`grad J = E[ sum_t grad log pi_theta(a_t|s_t) * Phi_t ]`. Five valid choices of `Phi_t`:
|
||||
full return, reward-to-go, reward-to-go minus a baseline, `Q^pi`, and `A^pi`. All share an
|
||||
expectation and differ in variance. Two rules get you from the first to the last:
|
||||
- **Don't let the past distract you** — drop rewards obtained before the action. Those terms
|
||||
had zero mean and nonzero variance: pure noise.
|
||||
- **Baselines** — by the EGLP lemma, any state-only `b(s)` can be added or subtracted freely.
|
||||
The standard choice is `V^pi(s_t)`, learned by MSE regression onto reward-to-go. (ch09)
|
||||
|
||||
### The policy-gradient loss is not a loss function
|
||||
Its data distribution depends on the parameters, and it does not measure performance even in
|
||||
expectation. Only at the current parameters, with data from those parameters, does it have
|
||||
the negative gradient of performance. **You can send it to negative infinity while performance
|
||||
craters, and it usually will. Only average return means anything.** (ch09)
|
||||
|
||||
### Broken RL code almost always fails silently
|
||||
It runs fine; the agent just never learns. Usually something is computed with the wrong
|
||||
equation, on the wrong distribution, or piped to the wrong place. **If it doesn't work, assume
|
||||
there's a bug** before touching hyperparameters. Debug by measuring everything and reading the
|
||||
code critically. The archetype is one missing `squeeze`: a `[N]` vs `[N,1]` shape mismatch is
|
||||
broadcast-compatible, raises nothing, and silently turns the Bellman backup into an `[N,N]`
|
||||
matrix. (ch10, ch12)
|
||||
|
||||
### Learn by doing
|
||||
Write your own implementations, **shortest correct version** of each, **simplest algorithms
|
||||
first**. VPG, DQN, A2C, PPO, DDPG, roughly in that order; ~250-300 lines each. Single-threaded
|
||||
before parallel. **Iterate fast in simple environments — under 5 minutes turnaround at the
|
||||
debug stage.** Do not attempt Atari or Humanoid before the toy task works. Read papers for
|
||||
their ablations and supplementary material, but **do not overfit to paper details** (the
|
||||
original DDPG's architecture, init scheme and batch norm are not strictly necessary) **or to
|
||||
existing implementations** (their abstractions serve reuse, not your single use case). (ch10)
|
||||
|
||||
### Rigor: four standards
|
||||
1. **Fair comparisons** — tune the baseline as hard as your method; never handicap it.
|
||||
2. **Remove stochasticity as a confounder** — at least 3 seeds, 10 or more to be thorough.
|
||||
Two seed groups can produce curves that look like different distributions.
|
||||
3. **High-integrity experiments** — launch fresh final runs and precommit to reporting them.
|
||||
Tuning produces hypotheses; final runs produce conclusions.
|
||||
4. **Check each claim separately** — ablate every design decision. (ch10)
|
||||
|
||||
### The safe-step family (on-policy)
|
||||
VPG takes an unconstrained gradient step, so a single bad step can collapse performance.
|
||||
**TRPO** constrains the step in **KL-divergence between policies, not distance in parameter
|
||||
space**, then backtracking-line-searches until the exact constraint holds. **PPO** drops the
|
||||
constraint and instead **clips the objective so the policy gains nothing by moving far**,
|
||||
which is first-order, far simpler, and empirically at least as good. (ch14, ch15, ch16)
|
||||
|
||||
### The overestimation family (off-policy)
|
||||
DDPG amortizes the intractable continuous `max_a Q(s,a)` into a learned policy:
|
||||
`max_a Q(s,a) ~= Q(s, mu(s))`. It needs a **replay buffer** (licensed because the Bellman
|
||||
equation is indifferent to how data was collected) and **target networks** (because the target
|
||||
otherwise depends on the parameters being trained). Its failure mode is **Q-value
|
||||
overestimation, which the policy actively exploits**. **TD3** answers with clipped double-Q,
|
||||
delayed policy updates and target policy smoothing. **SAC** adds **entropy regularization**,
|
||||
making the explore-exploit trade-off an explicit coefficient `alpha`. (ch17, ch18, ch19)
|
||||
|
||||
---
|
||||
|
||||
## Chapter Index
|
||||
|
||||
| # | Title | Key Frameworks |
|
||||
|---|-------|----------------|
|
||||
| [ch01](chapters/ch01-introduction.md) | Introduction | The missing middle step, Code Design Philosophy |
|
||||
| [ch02](chapters/ch02-installation.md) | Installation | Install-then-verify, MuJoCo optionality |
|
||||
| [ch03](chapters/ch03-algorithm-lineup.md) | Algorithms: What's Included and Why | The two lineages, on/off-policy trade-off, code template |
|
||||
| [ch04](chapters/ch04-running-experiments.md) | Running Experiments | One flag per kwarg, ExperimentGrid, save-dir suffixes |
|
||||
| [ch05](chapters/ch05-experiment-outputs.md) | Experiment Outputs | Tools not files, watch-then-measure |
|
||||
| [ch06](chapters/ch06-plotting-results.md) | Plotting Results | `Performance` alias, prefix autocompletion, seed averaging |
|
||||
| [ch07](chapters/ch07-key-concepts-in-rl.md) | Part 1: Key Concepts in RL | MDPs, four value functions, Bellman equations, advantage |
|
||||
| [ch08](chapters/ch08-kinds-of-rl-algorithms.md) | Part 2: Kinds of RL Algorithms | Taxonomy, model bias, policy-opt vs Q-learning |
|
||||
| [ch09](chapters/ch09-intro-to-policy-optimization.md) | Part 3: Intro to Policy Optimization | Log-derivative trick, EGLP lemma, reward-to-go, baselines |
|
||||
| [ch10](chapters/ch10-spinning-up-as-a-researcher.md) | Spinning Up as a Deep RL Researcher | Learn by doing, three idea frames, four rigor standards |
|
||||
| [ch11](chapters/ch11-key-papers-in-deep-rl.md) | Key Papers in Deep RL | 13-section topic map |
|
||||
| [ch12](chapters/ch12-exercises.md) | Exercises | Problem Set 1 and 2, the silent DDPG bug |
|
||||
| [ch13](chapters/ch13-benchmarks.md) | Benchmarks | The parity disclosure, family-specific metrics |
|
||||
| [ch14](chapters/ch14-vpg.md) | Vanilla Policy Gradient | The six-step loop |
|
||||
| [ch15](chapters/ch15-trpo.md) | Trust Region Policy Optimization | KL trust region, line search, conjugate gradient |
|
||||
| [ch16](chapters/ch16-ppo.md) | Proximal Policy Optimization | PPO-Clip, KL early stopping |
|
||||
| [ch17](chapters/ch17-ddpg.md) | Deep Deterministic Policy Gradient | MSBE, replay buffers, target networks, polyak |
|
||||
| [ch18](chapters/ch18-td3.md) | Twin Delayed DDPG | Clipped double-Q, delayed updates, target smoothing |
|
||||
| [ch19](chapters/ch19-sac.md) | Soft Actor-Critic | Entropy regularization, reparameterization, squashed Gaussian |
|
||||
| [ch20](chapters/ch20-logger-and-utilities.md) | Logger, MPI Tools and Run Utils | EpochLogger pattern, MPI PyTorch order |
|
||||
|
||||
## Topic Index
|
||||
|
||||
- **Advantage function** ch07, ch09, ch14
|
||||
- **Baselines** ch09
|
||||
- **Bellman equations** ch07, ch17
|
||||
- **Benchmarks / parity** ch13, ch01
|
||||
- **Clipped double-Q** ch18, ch19
|
||||
- **Continuous action spaces** ch07, ch17
|
||||
- **Debugging / silent failure** ch10, ch12
|
||||
- **DDPG** ch17, ch03, ch08
|
||||
- **Entropy regularization** ch19
|
||||
- **Exploration vs exploitation** ch14, ch17, ch19
|
||||
- **GAE** ch09, ch14
|
||||
- **Installation** ch02
|
||||
- **KL divergence / trust region** ch15, ch16
|
||||
- **Logging** ch20, ch05
|
||||
- **MDPs** ch07
|
||||
- **Model-based RL** ch08
|
||||
- **MPI / parallelization** ch20, ch02, ch04
|
||||
- **MSBE** ch17
|
||||
- **Off-policy** ch03, ch08, ch17
|
||||
- **On-policy** ch03, ch08, ch14
|
||||
- **Papers / literature** ch11, ch10
|
||||
- **Plotting** ch06, ch13
|
||||
- **Policies (categorical, Gaussian, squashed)** ch07, ch19
|
||||
- **Policy gradient derivation** ch09
|
||||
- **PPO** ch16, ch03
|
||||
- **Q-learning** ch08, ch07
|
||||
- **Replay buffer** ch17
|
||||
- **Reparameterization trick** ch19, ch10
|
||||
- **Research process / rigor** ch10, ch13
|
||||
- **Reward-to-go** ch09
|
||||
- **Running experiments** ch04, ch05
|
||||
- **SAC** ch19, ch03, ch08
|
||||
- **Seeds / variance** ch10, ch13, ch04
|
||||
- **Target networks / polyak** ch17, ch18
|
||||
- **TD3** ch18, ch12
|
||||
- **TRPO** ch15, ch03
|
||||
- **Value functions** ch07, ch09
|
||||
- **VPG** ch14, ch09
|
||||
|
||||
## Supporting Files
|
||||
|
||||
- [glossary.md](glossary.md) · [patterns.md](patterns.md) · [cheatsheet.md](cheatsheet.md)
|
||||
|
||||
## Scope & Limits
|
||||
|
||||
Covers the Spinning Up documentation only, as of the January 2020 PyTorch update. It does
|
||||
**not** cover: DQN and the discrete-action value-learning family (referenced, never
|
||||
implemented here), recurrent or convolutional architectures, partially-observed settings,
|
||||
model-based implementations, exploration/meta-RL/hierarchy beyond ch11's reading list, or any
|
||||
deep RL work after early 2020. The six implementations are educational; ch13 says which are
|
||||
research-grade. For topics beyond this source, I say so rather than improvising.
|
||||
|
||||
---
|
||||
|
||||
*Compiled from OpenAI's Spinning Up in Deep RL documentation (MIT, Copyright (c) 2018 OpenAI),
|
||||
primarily developed by Joshua Achiam. Structured study notes, not a reproduction of the source.*
|
||||
|
|
@ -0,0 +1,102 @@
|
|||
# Chapter 1: Introduction
|
||||
|
||||
## Core Idea
|
||||
Spinning Up exists to be the missing middle step between "I want to work on deep RL"
|
||||
and "I can implement and evaluate a deep RL algorithm" — the field has no standard
|
||||
textbook, papers omit the design details that decide whether code works, and production
|
||||
RL libraries hide the algorithm inside framework abstractions.
|
||||
|
||||
## Frameworks Introduced
|
||||
- **The missing middle step**: the gap Spinning Up fills, between high-level awareness
|
||||
(what topics exist, why they matter) and the ability to transmute an algorithm into code.
|
||||
- When to use: deciding whether a resource teaches you deep RL or merely describes it.
|
||||
- How: judge a resource on whether it closes the theory-to-code gap. Papers give theory
|
||||
and omit details; libraries give code that hides the algorithm. Neither closes it alone.
|
||||
|
||||
- **Code Design Philosophy** (the four rules the implementations obey):
|
||||
- **As simple as possible while still being reasonably good.** Not state of the art;
|
||||
good enough to reach roughly the intended performance.
|
||||
- **Highly consistent with each other**, so understanding one makes the next painless.
|
||||
- **Almost completely self-contained** — virtually no shared code between algorithms
|
||||
except logging, saving, loading and MPI utilities. You can study one algorithm without
|
||||
following a chain of dependencies.
|
||||
- **Patterned to come as close to pseudocode as possible**, minimizing the theory-code gap.
|
||||
- How: when reading or writing RL code, prefer the version you can read top to bottom
|
||||
over the version that reuses the most code. Reuse is an engineering virtue that is a
|
||||
pedagogical cost.
|
||||
|
||||
- **Minimize tricks and minimize differences**: the implementations deliberately omit
|
||||
tricks present in the original papers (e.g. the regularization terms in the original
|
||||
Soft Actor-Critic code, observation normalization in all algorithms) and deliberately
|
||||
remove gratuitous differences between similar algorithms (DDPG, TD3 and SAC all run
|
||||
gradient updates after fixed intervals of environment interaction, so they compare cleanly).
|
||||
- When to use: any time you are comparing two algorithms and want the comparison to be
|
||||
about the algorithms rather than about their implementation choices.
|
||||
|
||||
## Key Concepts
|
||||
- **Deep RL**: reinforcement learning (learning to solve tasks by trial and error) combined
|
||||
with deep learning.
|
||||
- **Reasonably good**: achieves roughly the intended performance but does not necessarily
|
||||
match the best reported results in the literature on every task.
|
||||
- **Maintenance mode**: Spinning Up's current status. Breaking bugs get repaired; no major
|
||||
new features are planned.
|
||||
- **The Rosetta Stone goal**: the motivation for the January 2020 PyTorch update — the same
|
||||
algorithm expressed in two neural network libraries so the algorithm is separable from the
|
||||
library.
|
||||
|
||||
## Mental Models
|
||||
- Think of an RL library's abstraction layer as a **tax on learning**: good for code reuse
|
||||
between algorithms, unnecessary if you are writing one algorithm for one use case.
|
||||
- Use "**can I read this algorithm without opening another file?**" as the test of whether
|
||||
an implementation is written to be learned from.
|
||||
- Think of the resource landscape as **two failure modes**: papers that obscure key design
|
||||
details, and public implementations that are hard to read. Spinning Up targets both.
|
||||
|
||||
## Anti-patterns
|
||||
- **Using Spinning Up's implementations for scientific benchmarking comparisons**: they are
|
||||
"reasonably good," not best-reported. See ch13 for which ones are at parity (DDPG, TD3, SAC)
|
||||
and which are not (VPG, TRPO, PPO). Use OpenAI Baselines for TRPO/PPO research comparisons.
|
||||
- **Assuming an educational implementation is a research implementation**: the omitted tricks
|
||||
(observation normalization, normalized value regression targets) are exactly what separates
|
||||
the two.
|
||||
- **Treating deep RL as engineering-only**: the material explicitly serves people from
|
||||
professions with no connection to engineering or computer science who nonetheless need to
|
||||
make informed decisions about the technology.
|
||||
|
||||
## Worked Example
|
||||
Support history, read as a record of what the community actually asked for after release:
|
||||
|
||||
| Date | Event |
|
||||
|------|-------|
|
||||
| Nov 8, 2018 | Initial release, followed by three weeks of high-bandwidth support |
|
||||
| April 2019 | Six-month internal review of community feedback |
|
||||
| Jan 2020 | The PyTorch update ships |
|
||||
| Future | No major updates planned |
|
||||
|
||||
The April 2019 review surfaced exactly three requests, in priority order:
|
||||
1. **Implementations in other neural network libraries** — enough people had written their
|
||||
own PyTorch ports (Fired Up, Spinning Up Basic, Torching Up) that a "Rosetta Stone for
|
||||
deep RL" became the top priority. This one shipped.
|
||||
2. **Open source RL environments** (e.g. PyBullet) for benchmarks, examples and exercises,
|
||||
to avoid the proprietary MuJoCo dependency. Did not ship.
|
||||
3. **More algorithms**, especially Deep Q-Networks. Did not ship.
|
||||
|
||||
The lesson for anyone building an educational resource: the community's top request was not
|
||||
more content, it was the same content expressed in the framework they already use.
|
||||
|
||||
## Key Takeaways
|
||||
1. The barrier to entry in deep RL is not intelligence or math; it is that theory and code
|
||||
are documented in separate places and neither is written to connect to the other.
|
||||
2. Self-contained beats DRY when the reader is trying to learn the algorithm.
|
||||
3. Consistency across implementations is a teaching feature: differences that remain are
|
||||
real algorithmic differences.
|
||||
4. "Reasonably good" is a deliberate, stated performance target — check ch13 before citing
|
||||
any Spinning Up number as a benchmark.
|
||||
5. Deep RL is central to AI safety work in OpenAI's framing; the resource exists partly as
|
||||
a recruiting and capability-building pipeline for that.
|
||||
|
||||
## Connects To
|
||||
- **Ch 3**: which algorithms were chosen and why, and the code template they all share.
|
||||
- **Ch 10**: the essay on becoming a researcher — the practice counterpart to this chapter's
|
||||
philosophy.
|
||||
- **Ch 13**: the benchmark numbers that qualify "reasonably good."
|
||||
|
|
@ -0,0 +1,88 @@
|
|||
# Chapter 2: Installation
|
||||
|
||||
## Core Idea
|
||||
Spinning Up needs Python 3, OpenAI Gym and OpenMPI, on Linux or OSX; MuJoCo is optional
|
||||
but is the de facto benchmarking standard, and everything in the book works without it on
|
||||
the free Classic Control and Box2D environments.
|
||||
|
||||
## Frameworks Introduced
|
||||
- **Install then immediately verify with a real training run**: the install is not confirmed
|
||||
by a successful `pip install`, it is confirmed by an agent visibly learning.
|
||||
- When to use: every fresh environment, before debugging anything else.
|
||||
- How: run the install test below, then watch the policy and plot the curve. Three
|
||||
commands, three different subsystems (training, rendering, plotting) verified.
|
||||
|
||||
## Key Concepts
|
||||
- **OpenMPI**: the message-passing library used for the parallelized (on-policy) algorithms.
|
||||
Installed with `apt-get install libopenmpi-dev` on Ubuntu, `brew install openmpi` on Mac.
|
||||
- **MuJoCo**: a proprietary physics engine; free to trial and free for full-time students,
|
||||
otherwise paid. The de facto standard for benchmarking continuous-control deep RL.
|
||||
- **mujoco-py**: the Python bindings that let Gym use MuJoCo.
|
||||
- **Classic Control / Box2D**: Gym environment families that are completely free and
|
||||
sufficient to get started.
|
||||
- **Editable install**: `pip install -e .` from the cloned repo, so edits to the algorithm
|
||||
files take effect without reinstalling — the intended workflow for a repo you are meant
|
||||
to read and modify.
|
||||
|
||||
## Code Examples
|
||||
Environment and dependencies:
|
||||
|
||||
```bash
|
||||
conda create -n spinningup python=3.6
|
||||
conda activate spinningup
|
||||
|
||||
# Ubuntu
|
||||
sudo apt-get update && sudo apt-get install libopenmpi-dev
|
||||
# Mac OS X (requires Homebrew)
|
||||
brew install openmpi
|
||||
|
||||
git clone https://github.com/openai/spinningup.git
|
||||
cd spinningup
|
||||
pip install -e .
|
||||
```
|
||||
|
||||
Verify the install (roughly 10 minutes; leave it running and keep reading):
|
||||
|
||||
```bash
|
||||
python -m spinup.run ppo --hid "[32,32]" --env LunarLander-v2 \
|
||||
--exp_name installtest --gamma 0.999
|
||||
python -m spinup.run test_policy data/installtest/installtest_s0
|
||||
python -m spinup.run plot data/installtest/installtest_s0
|
||||
```
|
||||
|
||||
Optional MuJoCo, after following the mujoco-py README and obtaining a license:
|
||||
|
||||
```bash
|
||||
pip install gym[mujoco,robotics]
|
||||
python -m spinup.run ppo --hid "[32,32]" --env Walker2d-v2 --exp_name mujocotest
|
||||
```
|
||||
|
||||
## Reference Tables
|
||||
|
||||
| Requirement | Status | Notes |
|
||||
|-------------|--------|-------|
|
||||
| Python 3 | Required | Anaconda recommended; the docs pin 3.6 |
|
||||
| OpenAI Gym | Required | Installed by `pip install -e .`, MuJoCo envs excluded |
|
||||
| OpenMPI | Required | Needed for the MPI-parallelized on-policy algorithms |
|
||||
| Linux / OSX | Required | Windows unsupported and untested; one community workaround exists |
|
||||
| MuJoCo | Optional | Proprietary, licensed; preferred because of benchmarking convention |
|
||||
|
||||
## Anti-patterns
|
||||
- **Skipping the install test** and then debugging an algorithm that was never installed
|
||||
correctly. The install test costs ten unattended minutes.
|
||||
- **Treating a MuJoCo licence as a prerequisite for learning RL**: Classic Control and Box2D
|
||||
are free and adequate for everything up to benchmarking.
|
||||
- **Fighting package management ad hoc**: the docs go out of their way to point at conda
|
||||
explainers, because "I just installed this thing but it says it's not found" is the
|
||||
predicted failure, not an unusual one.
|
||||
|
||||
## Key Takeaways
|
||||
1. Install is Python 3 + Gym + OpenMPI, plus optional MuJoCo; Linux or OSX only.
|
||||
2. `pip install -e .` (editable) because you are expected to modify the code.
|
||||
3. Verification means an actual short PPO run, a rendered policy and a plot.
|
||||
4. Not having MuJoCo blocks benchmarking, not learning.
|
||||
|
||||
## Connects To
|
||||
- **Ch 4**: `python -m spinup.run` is the entry point every later chapter uses.
|
||||
- **Ch 13**: the benchmarks that make MuJoCo the convention.
|
||||
- **Ch 20**: the MPI utilities that OpenMPI enables.
|
||||
|
|
@ -0,0 +1,101 @@
|
|||
# Chapter 3: Algorithms — What's Included and Why
|
||||
|
||||
## Core Idea
|
||||
The six algorithms (VPG, TRPO, PPO, DDPG, TD3, SAC) are not a survey; they are two
|
||||
lineages of ideas, each starting from a foundational algorithm and progressively fixing
|
||||
its defining weakness, culminating in PPO and SAC.
|
||||
|
||||
## Frameworks Introduced
|
||||
- **The two lineages**:
|
||||
- **On-policy line: VPG to TRPO to PPO.** Each step buys back sample efficiency without
|
||||
giving up the stability that comes from directly optimizing the objective you care about.
|
||||
- **Off-policy line: DDPG to TD3 and SAC.** Each step mitigates the brittleness that comes
|
||||
from optimizing a Bellman self-consistency condition instead of performance itself.
|
||||
- When to use: placing any new algorithm you read about. Ask which line it is on and which
|
||||
weakness of its predecessor it claims to fix.
|
||||
|
||||
- **The central trade-off the lineup exposes**: on-policy algorithms *directly optimize the
|
||||
objective you care about* — policy performance — and it works out mathematically that
|
||||
you need on-policy data to compute the updates, so they cannot reuse old data. Off-policy
|
||||
algorithms exploit the Bellman optimality equations, which hold for *any* transition data,
|
||||
so they reuse everything — but satisfying Bellman's equations well carries **no guarantee**
|
||||
of good policy performance.
|
||||
- How: choose the family by which risk you can afford. Stability with a sample budget, or
|
||||
sample efficiency with a tuning budget.
|
||||
|
||||
- **The standard two-file code template**: every implementation splits into an *algorithm
|
||||
file* (experience buffer class, then one function that runs the algorithm) and a *core
|
||||
file* (utilities, the actor-critic constructors, the MLP actor-critic).
|
||||
- When to use: reading any Spinning Up algorithm, or structuring your own.
|
||||
|
||||
## Key Concepts
|
||||
- **On-policy**: each update uses only data collected by the most recent version of the policy.
|
||||
- **Off-policy**: each update can use data collected at any point in training, however the
|
||||
agent was exploring at the time.
|
||||
- **MLP actor-critic**: all six use non-recurrent multi-layer-perceptron actor-critics, which
|
||||
makes them suitable for fully-observed, non-image-based environments (e.g. Gym MuJoCo) and
|
||||
unsuitable, as shipped, for partial observability or pixels.
|
||||
- **Sample efficiency**: how much environment interaction is needed to reach a performance level.
|
||||
- **Deterministic policy gradients**: the theory (2014) that made DDPG possible — much younger
|
||||
than the policy gradient theory behind VPG, whose core elements go back to the late 80s.
|
||||
|
||||
## Reference Tables
|
||||
|
||||
| Algorithm | Family | Policy | Action spaces | Parallel (MPI) | PyTorch | TF1 |
|
||||
|-----------|--------|--------|---------------|----------------|---------|-----|
|
||||
| VPG | On-policy | Stochastic | Discrete + continuous | Yes | Yes | Yes |
|
||||
| TRPO | On-policy | Stochastic | Discrete + continuous | Yes | No | Yes |
|
||||
| PPO | On-policy | Stochastic | Discrete + continuous | Yes | Yes | Yes |
|
||||
| DDPG | Off-policy | Deterministic | Continuous only | No | Yes | Yes |
|
||||
| TD3 | Off-policy | Deterministic | Continuous only | No | Yes | Yes |
|
||||
| SAC | Off-policy | Stochastic | Continuous only (as shipped) | No | Yes | Yes |
|
||||
|
||||
## Code Examples
|
||||
The PyTorch algorithm function, in order — the template every implementation follows:
|
||||
|
||||
```
|
||||
1) Logger setup
|
||||
2) Random seed setting
|
||||
3) Environment instantiation
|
||||
4) Build the actor-critic module via the `actor_critic` function passed in as an argument
|
||||
5) Instantiate the experience buffer
|
||||
6) Set up callable loss functions that also return algorithm-specific diagnostics
|
||||
7) Make PyTorch optimizers
|
||||
8) Set up model saving through the logger
|
||||
9) Set up an update function: one epoch of optimization, or one step of descent
|
||||
10) Main loop: a) run the agent in the environment
|
||||
b) periodically update parameters per the algorithm's main equations
|
||||
c) log key performance metrics and save the agent
|
||||
```
|
||||
|
||||
The TF1 version is the same shape with four graph-construction steps inserted (placeholders,
|
||||
actor-critic graph, loss/diagnostic graph, training ops) and a session step.
|
||||
|
||||
## Mental Models
|
||||
- Read the lineup as **"progressions of ideas from the recent history of the field"**, not as
|
||||
a menu. VPG predates deep RL entirely; DDPG's theory is from 2014.
|
||||
- Treat **PPO and SAC as the two defaults** — they are close to state of the art on
|
||||
reliability and sample efficiency among policy-learning algorithms.
|
||||
- Think of the `actor_critic` argument as the seam: the algorithm function is fixed, the
|
||||
network construction is injected. That is also where the ch12 silent bug lives.
|
||||
|
||||
## Anti-patterns
|
||||
- **Reaching for DDPG on a discrete action space.** DDPG, TD3 and the shipped SAC are
|
||||
continuous-only. The max over actions is why (see ch17).
|
||||
- **Expecting these to work on Atari or partially-observed tasks as shipped**: MLP,
|
||||
non-recurrent, non-image.
|
||||
- **Picking an algorithm by recency**: TD3 and SAC were published roughly concurrently; SAC
|
||||
is not a successor to TD3, it is a parallel branch that borrowed the clipped double-Q trick.
|
||||
|
||||
## Key Takeaways
|
||||
1. Two families, one trade-off: directly optimize performance (on-policy, stable, sample-hungry)
|
||||
or exploit Bellman (off-policy, sample-efficient, no performance guarantee).
|
||||
2. PPO and SAC are the practical end points of each line.
|
||||
3. Continuous-only for the whole off-policy line, as implemented here.
|
||||
4. Every implementation is two files and the same ten-step function; learn the template once.
|
||||
5. TRPO has no PyTorch implementation in Spinning Up.
|
||||
|
||||
## Connects To
|
||||
- **Ch 8**: the full taxonomy this lineup is a slice of.
|
||||
- **Ch 14-19**: one chapter per algorithm, in lineage order.
|
||||
- **Ch 12**: the exercises that make the on-policy/off-policy distinction concrete.
|
||||
|
|
@ -0,0 +1,133 @@
|
|||
# Chapter 4: Running Experiments
|
||||
|
||||
## Core Idea
|
||||
`spinup/run.py` turns every keyword argument of every algorithm into a command-line flag,
|
||||
and turns a list of values for a flag into a series of experiments — so hyperparameter
|
||||
sweeps and multi-seed runs are one command, not a script.
|
||||
|
||||
## Frameworks Introduced
|
||||
- **One flag per kwarg**: if `kwarg` is a valid keyword argument of an algorithm function,
|
||||
`--kwarg` sets it. `python -m spinup.run [algo] --help` prints the docstring readout.
|
||||
- How: `python -m spinup.run [algo name] [experiment flags]`.
|
||||
|
||||
- **Multiple values means multiple experiments**: providing more than one value for an
|
||||
argument launches one experiment per combination, **in series**. They do not run in
|
||||
parallel because a single experiment already soaks up enough resources that concurrency
|
||||
buys no speedup.
|
||||
- When to use: seed sweeps (the ch10 rigor requirement) and hyperparameter ablations.
|
||||
|
||||
- **ExperimentGrid**: the in-script equivalent, based on but simpler than rllab's
|
||||
VariantGenerator. `spinup.run` uses one under the hood.
|
||||
- How: `eg.add(param_name, values, shorthand, in_name)` then `eg.run(thunk, **run_kwargs)`.
|
||||
`in_name` forces a parameter into the experiment name even when it does not vary.
|
||||
- Difference from the CLI: no shortcut kwargs — you must write `ac_kwargs:hidden_sizes`,
|
||||
not `hid`.
|
||||
|
||||
- **Shorthand-driven save directories**: results land in
|
||||
`data_dir/[outer_prefix]exp_name[suffix]/[inner_prefix]exp_name[suffix]_s[seed]`.
|
||||
The suffix encodes only the hyperparameters that *differ across the launched experiments*,
|
||||
and never the seed — so runs that differ only by seed group into the same folder, which is
|
||||
exactly what the plotter needs to average over seeds.
|
||||
|
||||
## Key Concepts
|
||||
- **`--env` / `--env_name`**: a Gym environment name, converted internally into the `env_fn`
|
||||
callable every algorithm actually takes.
|
||||
- **`--hid` / `--act`**: shortcut flags for `ac_kwargs:hidden_sizes` and `ac_kwargs:activation`,
|
||||
valid for all current algorithms.
|
||||
- **`--cpu` / `--num_cpu`**: launch with this many MPI-connected processes; `auto` uses all
|
||||
available. Raises an error for algorithms that do not support parallelization.
|
||||
- **`--dt` / `--datestamp`**: put timestamps in the save directory names.
|
||||
- **User-supplied shorthand**: square brackets after a flag, e.g. `--hid[h]`, control the
|
||||
directory-name abbreviation. Without one, a shorthand is derived automatically
|
||||
(`clip_ratio` becomes `cli`).
|
||||
- **`eval()` passthrough**: flag values pass through `eval()` before use, so you can name
|
||||
functions and objects directly, e.g. `--act torch.nn.ELU`.
|
||||
- **Dict kwargs**: `--key:v1 value_1 --key:v2 value_2` instead of `--key dict(v1=..., v2=...)`.
|
||||
|
||||
## Code Examples
|
||||
The quickstart, with every flag doing something distinct:
|
||||
|
||||
```bash
|
||||
python -m spinup.run ppo --exp_name ppo_ant --env Ant-v2 --clip_ratio 0.1 0.2 \
|
||||
--hid[h] [32,32] [64,32] --act torch.nn.Tanh --seed 0 10 20 --dt \
|
||||
--data_dir path/to/data
|
||||
```
|
||||
|
||||
Choosing the backend explicitly:
|
||||
|
||||
```bash
|
||||
python -m spinup.run ppo_pytorch --env Walker2d-v2 --exp_name walker
|
||||
python -m spinup.run ppo_tf1 --env Walker2d-v2 --exp_name walker
|
||||
# bare `ppo` reads spinup/user_config.py for the default backend
|
||||
```
|
||||
|
||||
From a script:
|
||||
|
||||
```python
|
||||
from spinup import ppo_pytorch as ppo
|
||||
import gym
|
||||
env_fn = lambda: gym.make('LunarLander-v2')
|
||||
ac_kwargs = dict(hidden_sizes=[64, 64])
|
||||
logger_kwargs = dict(output_dir='path/to/output_dir', exp_name='experiment_name')
|
||||
ppo(env_fn=env_fn, ac_kwargs=ac_kwargs, steps_per_epoch=5000, epochs=250,
|
||||
logger_kwargs=logger_kwargs)
|
||||
```
|
||||
|
||||
ExperimentGrid, from `spinup/examples/pytorch/bench_ppo_cartpole.py`:
|
||||
|
||||
```python
|
||||
from spinup.utils.run_utils import ExperimentGrid
|
||||
from spinup import ppo_pytorch
|
||||
import torch
|
||||
|
||||
eg = ExperimentGrid(name='ppo-pyt-bench')
|
||||
eg.add('env_name', 'CartPole-v0', '', True)
|
||||
eg.add('seed', [10*i for i in range(args.num_runs)])
|
||||
eg.add('epochs', 10)
|
||||
eg.add('steps_per_epoch', 4000)
|
||||
eg.add('ac_kwargs:hidden_sizes', [(32,), (64,64)], 'hid')
|
||||
eg.add('ac_kwargs:activation', [torch.nn.Tanh, torch.nn.ReLU], '')
|
||||
eg.run(ppo_pytorch, num_cpu=args.cpu)
|
||||
```
|
||||
|
||||
## Worked Example
|
||||
`python -m spinup.run ddpg_tf1 --env Hopper-v2 --hid[h] [300] [128,128] --act tf.nn.tanh tf.nn.relu`
|
||||
|
||||
Two `hid` values times two `act` values is four experiments, run in series, producing four
|
||||
suffixes:
|
||||
|
||||
```
|
||||
_h128-128_ac-actrelu
|
||||
_h128-128_ac-acttanh
|
||||
_h300_ac-actrelu
|
||||
_h300_ac-acttanh
|
||||
```
|
||||
|
||||
`h` came from the user-supplied `[h]`; `ac-act` was derived automatically from the true flag
|
||||
name `ac_kwargs:activation`. Add `--seed 0 10 20` and you get twelve runs in four folders —
|
||||
three seeds grouped per configuration, which is the grouping the plotter averages over.
|
||||
|
||||
## Anti-patterns
|
||||
- **Running the per-algorithm files directly** (`spinup/algos/BACKEND/ALGO/ALGO.py`). The
|
||||
command-line support there is vestigial, takes a different argument set, and is explicitly
|
||||
not the recommended way to run experiments.
|
||||
- **Using ZShell without escaping square brackets**: ZShell treats them as special characters
|
||||
and Spinning Up uses them for both list values and shorthands.
|
||||
- **One seed.** RL algorithms have high variance; the flag exists so you use it (ch10 asks
|
||||
for at least 3, ideally 10 or more).
|
||||
- **Expecting parallel experiment launches.** Sweeps run in series by design.
|
||||
|
||||
## Key Takeaways
|
||||
1. Every kwarg is a flag; every list of flag values is a sweep.
|
||||
2. Sweeps run in series; MPI parallelism is per-experiment via `--cpu`, and only for
|
||||
algorithms that support it.
|
||||
3. Save-directory suffixes encode only the varying hyperparameters, never the seed — that
|
||||
grouping is what makes multi-seed plotting work.
|
||||
4. `ExperimentGrid` is the scripted form and loses only the shortcut kwargs.
|
||||
5. Values go through `eval()`, so activations and other objects can be named on the command line.
|
||||
|
||||
## Connects To
|
||||
- **Ch 5**: what the directories those flags create actually contain.
|
||||
- **Ch 6**: the plotter, which consumes this directory structure and its autocompletion.
|
||||
- **Ch 10**: why multi-seed runs are a rigor requirement, not a convenience.
|
||||
- **Ch 20**: `ExperimentGrid`, `call_experiment` and `setup_logger_kwargs` in Run Utils.
|
||||
|
|
@ -0,0 +1,97 @@
|
|||
# Chapter 5: Experiment Outputs
|
||||
|
||||
## Core Idea
|
||||
Every run saves four things — a config record, a TSV of training metrics, the trained
|
||||
model, and a pickled copy of the environment — and only one of them (`config.json`) is
|
||||
ever meant to be read by hand.
|
||||
|
||||
## Frameworks Introduced
|
||||
- **Tools, not files**: `test_policy` loads from `pyt_save/` or `tf1_save/`, the plotter
|
||||
interprets `progress.txt`. Those are the correct interfaces. `config.json` is the only
|
||||
file you should ever open yourself, and only to remember what you ran.
|
||||
- When to use: any time you are tempted to parse a save directory manually.
|
||||
|
||||
- **Watch, then measure**: `test_policy` renders the agent so you can see behavior that no
|
||||
scalar metric would have shown you; `-nr` drops rendering when you only want the numbers.
|
||||
|
||||
## Key Concepts
|
||||
- **`progress.txt`**: tab-separated records of every metric the logger recorded, e.g. `Epoch`,
|
||||
`AverageEpRet`.
|
||||
- **`config.json`**: as-complete-as-possible dict of the args and kwargs used to launch the run.
|
||||
Record-keeping only — launching an experiment from a config file is not supported.
|
||||
- **`vars.pkl`**: algorithm state; currently used only to save a copy of the environment.
|
||||
- **`pyt_save/model.pt`**: a pickled PyTorch `nn.Module`; loading restores an ActorCritic
|
||||
object with an `act` method.
|
||||
- **`tf1_save/`**: `variables/`, `model_info.pkl` (key-to-tensor-name map) and `saved_model.pb`.
|
||||
Renamed from `simple_save/` on 2020-01-30.
|
||||
- **`DEFAULT_DATA_DIR`**: set in `spinup/user_config.py`; defaults to `spinningup/data`.
|
||||
|
||||
## Reference Tables
|
||||
|
||||
| File | Contents |
|
||||
|------|----------|
|
||||
| `pyt_save/` | PyTorch only. Everything needed to restore the agent and value functions |
|
||||
| `tf1_save/` | TF1 only. SavedModel plus the key-to-tensor map |
|
||||
| `config.json` | The launch args and kwargs. Non-serializable values become strings |
|
||||
| `progress.txt` | TSV of logged metrics across training |
|
||||
| `vars.pkl` | Pickled environment copy; may be empty if the env cannot be pickled |
|
||||
|
||||
`test_policy` flags:
|
||||
|
||||
| Flag | Default | Does |
|
||||
|------|---------|------|
|
||||
| `-l L`, `--len=L` | 0 | Max episode length; 0 means no maximum |
|
||||
| `-n N`, `--episodes=N` | 100 | Number of test episodes |
|
||||
| `-nr`, `--norender` | off | Print returns and lengths only; much faster |
|
||||
| `-i I`, `--itr=I` | -1 | Which saved snapshot; -1 is latest (see below) |
|
||||
| `-d`, `--deterministic` | off | **SAC only.** Use the deterministic mean policy |
|
||||
|
||||
## Code Examples
|
||||
```bash
|
||||
python -m spinup.run test_policy path/to/output_directory
|
||||
```
|
||||
|
||||
When the environment failed to pickle and `test_policy` raises `AssertionError: Environment
|
||||
not found!`, rebuild it by hand:
|
||||
|
||||
```python
|
||||
from spinup.utils.test_policy import load_policy_and_env, run_policy
|
||||
import your_env
|
||||
_, get_action = load_policy_and_env('/path/to/output_directory')
|
||||
env = your_env.make()
|
||||
run_policy(env, get_action)
|
||||
```
|
||||
|
||||
To keep snapshots from many points in training (off by default — algorithms overwrite the
|
||||
most recent one), change the line present in every algorithm:
|
||||
|
||||
```python
|
||||
logger.save_state({'env': env}, None) # default: one snapshot, overwritten
|
||||
logger.save_state({'env': env}, epoch) # keep per-epoch snapshots
|
||||
```
|
||||
|
||||
and then set `save_freq` to something reasonable — at the default of 1 you flood the output
|
||||
directory with one folder per epoch.
|
||||
|
||||
## Anti-patterns
|
||||
- **Forgetting `-d` when evaluating SAC.** SAC trains a stochastic policy but the correct
|
||||
evaluation policy is the deterministic mean. Without the flag you are measuring the wrong
|
||||
thing, and the flag is used for no other algorithm.
|
||||
- **Expecting to resume training.** Spinning Up implementations have no way to resume a
|
||||
partially-trained agent.
|
||||
- **Trying to relaunch from `config.json`.** It is a record, not an input.
|
||||
- **Assuming `vars.pkl` has your environment.** Gym Box2D environments in older Gym versions
|
||||
are known to fail to pickle, leaving it empty.
|
||||
|
||||
## Key Takeaways
|
||||
1. Four artifacts per run; use the tools for three of them and read only `config.json` by hand.
|
||||
2. SAC evaluation requires `--deterministic`; every other algorithm ignores it.
|
||||
3. Multi-snapshot saving is a two-line change plus a `save_freq` you must set deliberately.
|
||||
4. Environment pickling can silently fail — the recovery path is rebuilding the env in Python.
|
||||
5. No resume support; a killed run is a lost run.
|
||||
|
||||
## Connects To
|
||||
- **Ch 4**: the flags that create these directories.
|
||||
- **Ch 6**: `progress.txt` is what the plotter reads.
|
||||
- **Ch 19**: why SAC's evaluation policy differs from its training policy.
|
||||
- **Ch 20**: the EpochLogger that writes all of this.
|
||||
|
|
@ -0,0 +1,79 @@
|
|||
# Chapter 6: Plotting Results
|
||||
|
||||
## Core Idea
|
||||
The plotter reads a directory prefix, autocompletes every matching run beneath it, averages
|
||||
across seeds by default, and resolves the pseudo-metric `Performance` to the *correct*
|
||||
performance measure for each algorithm family — so a fair on-policy versus off-policy
|
||||
comparison is one command.
|
||||
|
||||
## Frameworks Introduced
|
||||
- **`Performance` as a family-aware alias**: `Performance` is not a real output of any
|
||||
algorithm. The plotter resolves it per logdir to `AverageEpRet` for on-policy algorithms
|
||||
and `AverageTestEpRet` for off-policy ones.
|
||||
- When to use: always, unless you specifically want one raw column. It is the default `-y`.
|
||||
- Why it matters: on-policy performance is the average return of the batch just collected;
|
||||
off-policy performance is measured by separate deterministic test rollouts. Plotting the
|
||||
same raw column for both would compare two different quantities.
|
||||
|
||||
- **Prefix autocompletion**: logdirs are searched recursively and prefixes expand. Give the
|
||||
plotter `data/bench_algo` and it finds `bench_algo1` and `bench_algo2` with all their seeds.
|
||||
- How: name experiments with a shared prefix at launch and comparison becomes free.
|
||||
|
||||
- **Average by default, `--count` to disaggregate**: by default, y-values are averaged across
|
||||
all results sharing an `exp_name` — typically identical experiments differing only in seed.
|
||||
`--count` shows each curve separately.
|
||||
|
||||
## Key Concepts
|
||||
- **`--xaxis` / `-x`**: which column is the x-axis. Default `TotalEnvInteracts`, i.e. sample
|
||||
efficiency is the default framing, not wall-clock or epochs.
|
||||
- **`--value` / `-y`**: which columns to graph; multiple values produce multiple graphs.
|
||||
- **`--smooth S`**: average over a fixed window of width S. The book's own benchmark plots
|
||||
use a window of 11 epochs (ch13).
|
||||
- **`--legend` / `-l`**: overrides the automatic legend, which uses `exp_name` from
|
||||
`config.json`. One string per *matched* directory, which may exceed the number of logdir
|
||||
arguments you passed.
|
||||
- **`--select` / `--exclude`**: keep or drop curves whose logdir contains all of / any of
|
||||
these substrings. The way to narrow an over-eager autocomplete.
|
||||
|
||||
## Code Examples
|
||||
```bash
|
||||
python -m spinup.run plot [path/to/output_directory ...] [--legend [LEGEND ...]] \
|
||||
[--xaxis XAXIS] [--value [VALUE ...]] [--count] [--smooth S] \
|
||||
[--select [SEL ...]] [--exclude [EXC ...]]
|
||||
```
|
||||
|
||||
Comparing two algorithms across all their seeds, relying on autocompletion:
|
||||
|
||||
```
|
||||
data/
|
||||
bench_algo1/
|
||||
bench_algo1-seed0/
|
||||
bench_algo1-seed10/
|
||||
bench_algo2/
|
||||
bench_algo2-seed0/
|
||||
bench_algo2-seed10/
|
||||
```
|
||||
|
||||
```bash
|
||||
python spinup/utils/plot.py data/bench_algo
|
||||
```
|
||||
|
||||
## Anti-patterns
|
||||
- **Passing one legend string per logdir argument.** Autocompletion may match several
|
||||
directories per argument; you need one legend entry per match, unless you narrowed the
|
||||
set with `--select` / `--exclude`.
|
||||
- **Plotting `AverageEpRet` for an off-policy algorithm** and calling it performance — that
|
||||
is the behavior policy with exploration noise, not the evaluated policy. Use `Performance`.
|
||||
- **Heavy smoothing to make a result look clean.** Smoothing is a display parameter; it does
|
||||
not change what the seeds did, and ch10 asks you to show seed variance rather than hide it.
|
||||
|
||||
## Key Takeaways
|
||||
1. `Performance` is the correct default because it is algorithm-family aware.
|
||||
2. Prefix autocompletion plus shared `exp_name` prefixes is the intended comparison workflow.
|
||||
3. Seed averaging is on by default; `--count` reveals the individual curves that average hides.
|
||||
4. The default x-axis is environment interactions — sample efficiency is the default question.
|
||||
|
||||
## Connects To
|
||||
- **Ch 5**: `progress.txt`, the plotter's input.
|
||||
- **Ch 4**: the suffix rules that group seeds into one folder for averaging.
|
||||
- **Ch 13**: the benchmark plots, produced with 10 seeds and an 11-epoch smoothing window.
|
||||
|
|
@ -0,0 +1,139 @@
|
|||
# Chapter 7: Part 1 — Key Concepts in RL
|
||||
|
||||
## Core Idea
|
||||
RL is the study of agents learning by trial and error; formally, the agent maximizes
|
||||
expected return J(pi) over trajectories, and almost every algorithm gets there through one
|
||||
of four value functions, all of which obey Bellman self-consistency equations.
|
||||
|
||||
## Frameworks Introduced
|
||||
- **The agent-environment interaction loop**: at every step the agent sees a (possibly
|
||||
partial) observation, chooses an action, and receives a reward. The environment changes
|
||||
because of the action, and may also change on its own.
|
||||
- How: name the five MDP pieces before writing any code — states S, actions A, reward
|
||||
function R, transition function P, start-state distribution rho_0.
|
||||
|
||||
- **The RL optimization problem**: `pi* = argmax_pi J(pi)`, where
|
||||
`J(pi) = E_{tau ~ pi}[R(tau)]` and the trajectory distribution is
|
||||
`P(tau|pi) = rho_0(s_0) * prod_t P(s_{t+1}|s_t,a_t) * pi(a_t|s_t)`.
|
||||
- When to use: as the definition every algorithm is trying to approximate. If an algorithm
|
||||
optimizes something else (a Bellman residual, a surrogate objective), that substitution
|
||||
is the source of its failure modes.
|
||||
|
||||
- **The four value functions**:
|
||||
- `V^pi(s)` — on-policy value: expected return starting in s and acting by pi forever.
|
||||
- `Q^pi(s,a)` — on-policy action-value: same, but take an arbitrary a first.
|
||||
- `V*(s)` — optimal value: expected return acting optimally from s.
|
||||
- `Q*(s,a)` — optimal action-value: take arbitrary a, then act optimally.
|
||||
- Two connections that come up constantly: `V^pi(s) = E_{a~pi}[Q^pi(s,a)]` and
|
||||
`V*(s) = max_a Q*(s,a)`.
|
||||
|
||||
- **The optimal-action shortcut**: if you have `Q*`, you get the optimal policy for free:
|
||||
`a*(s) = argmax_a Q*(s,a)`. There may be several maximizers, all optimal, but there is
|
||||
always an optimal policy that picks deterministically.
|
||||
- When to use: this is the entire justification for Q-learning as a family — learn `Q*`
|
||||
and the policy is a lookup. Ch17 explains what breaks when the argmax is over a
|
||||
continuous space.
|
||||
|
||||
- **Bellman equations**: "the value of your starting point is the reward you expect to get
|
||||
from being there, plus the value of wherever you land next."
|
||||
- On-policy: `V^pi(s) = E_{a~pi, s'~P}[r(s,a) + gamma V^pi(s')]`
|
||||
- Optimal: `V*(s) = max_a E_{s'~P}[r(s,a) + gamma V*(s')]`
|
||||
- The one crucial difference is the presence of the `max` over actions in the optimal form,
|
||||
reflecting that an agent free to choose must pick the highest-value action.
|
||||
- **Bellman backup** = the right-hand side, the reward-plus-next-value.
|
||||
|
||||
- **The advantage function**: `A^pi(s,a) = Q^pi(s,a) - V^pi(s)`. How much better taking a
|
||||
specific action is than randomly selecting one according to pi, assuming you follow pi
|
||||
afterwards.
|
||||
- When to use: whenever relative quality is what matters and absolute value is noise.
|
||||
Crucially important to policy gradient methods (ch9).
|
||||
|
||||
## Key Concepts
|
||||
- **State vs observation**: a state s is a complete description of the world; an observation
|
||||
o may omit information. Fully observed vs partially observed environments. Notation
|
||||
routinely writes s where o is technically correct.
|
||||
- **Action space**: the set of valid actions. **Discrete** (Atari, Go) versus **continuous**
|
||||
(robot control, real-valued vectors). The distinction has profound consequences: some
|
||||
algorithm families apply directly only to one case.
|
||||
- **Policy**: deterministic `a_t = mu(s_t)` or stochastic `a_t ~ pi(.|s_t)`. In deep RL these
|
||||
are **parameterized** — parameters theta or phi written as a subscript. "Policy" is often
|
||||
used interchangeably with "agent."
|
||||
- **Trajectory** (also **episode**, **rollout**): `tau = (s_0, a_0, s_1, a_1, ...)`.
|
||||
- **Finite-horizon undiscounted return**: `R(tau) = sum_{t=0}^{T} r_t`.
|
||||
- **Infinite-horizon discounted return**: `R(tau) = sum_{t=0}^{inf} gamma^t r_t`, with
|
||||
`gamma` in (0,1).
|
||||
- **MDP**: the 5-tuple `<S, A, R, P, rho_0>`. Markov property: transitions depend only on the
|
||||
most recent state and action, not on prior history.
|
||||
|
||||
## Code Examples
|
||||
A deterministic continuous-action policy is just an MLP:
|
||||
|
||||
```python
|
||||
pi_net = nn.Sequential(
|
||||
nn.Linear(obs_dim, 64), nn.Tanh(),
|
||||
nn.Linear(64, 64), nn.Tanh(),
|
||||
nn.Linear(64, act_dim)
|
||||
)
|
||||
obs_tensor = torch.as_tensor(obs, dtype=torch.float32)
|
||||
actions = pi_net(obs_tensor)
|
||||
```
|
||||
|
||||
**Categorical policies** (discrete actions) are built exactly like a classifier: observation
|
||||
in, layers, a final linear layer giving logits per action, softmax to probabilities. Sampling
|
||||
uses the framework's built-in categorical sampler. Log-likelihood is a vector index:
|
||||
`log pi_theta(a|s) = log [P_theta(s)]_a`.
|
||||
|
||||
**Diagonal Gaussian policies** (continuous actions) always have a network mapping observations
|
||||
to mean actions `mu_theta(s)`. The covariance is diagonal, so it is a vector, represented one
|
||||
of two ways:
|
||||
1. A single **state-independent** vector of log standard deviations — standalone parameters.
|
||||
*Spinning Up's VPG, TRPO and PPO do it this way.*
|
||||
2. A network `log sigma_theta(s)` mapping states to log standard deviations, optionally
|
||||
sharing layers with the mean network. *SAC does it this way, and SAC with state-independent
|
||||
log stds did not work (ch19).*
|
||||
|
||||
Log standard deviations, not standard deviations, because logs are free to range over
|
||||
(-inf, inf) while stds must be nonnegative, and unconstrained parameters are easier to train.
|
||||
Nothing is lost: exponentiate to recover.
|
||||
|
||||
Sampling: `a = mu_theta(s) + sigma_theta(s) * z` with `z ~ N(0, I)` (elementwise product).
|
||||
|
||||
Log-likelihood for a k-dimensional action:
|
||||
`log pi_theta(a|s) = -0.5 * ( sum_i [ (a_i - mu_i)^2 / sigma_i^2 + 2 log sigma_i ] + k log 2pi )`
|
||||
|
||||
## Mental Models
|
||||
- **A discount factor is two arguments in one**: intuitively, cash now beats cash later;
|
||||
mathematically, an infinite sum of rewards may not converge and is hard to work with.
|
||||
- **The formalism is starker than the practice.** Deep RL routinely sets up algorithms to
|
||||
optimize the *undiscounted* return while using discount factors when *estimating value
|
||||
functions*. Expect the line to be blurred in real code.
|
||||
- Treat **Q\* as a policy in disguise** and **V\* as a scoring function**: only the action-value
|
||||
form directly yields an action.
|
||||
- Ask of every value function: **is it time-dependent?** Unless stated otherwise, value
|
||||
functions mean infinite-horizon discounted return. Finite-horizon undiscounted value
|
||||
functions would need time as an argument — because what a state is worth depends on how
|
||||
much time is left.
|
||||
|
||||
## Anti-patterns
|
||||
- **Reading `s` in an equation as a true state.** In partially observed settings the action
|
||||
is conditioned on the observation; the notation is convention, not a claim.
|
||||
- **Assuming an algorithm ports across action-space types.** Moving a method between discrete
|
||||
and continuous spaces can require substantial rework, not a flag.
|
||||
- **Learning `V*` and expecting to act.** Without `Q*` or a model you cannot extract the
|
||||
action from the value.
|
||||
|
||||
## Key Takeaways
|
||||
1. The goal is always `argmax_pi E_{tau~pi}[R(tau)]`; everything else is machinery for
|
||||
approximating it.
|
||||
2. Four value functions, two of them optimal; `V^pi = E_a[Q^pi]` and `V* = max_a Q*` connect them.
|
||||
3. Bellman equations hold for all four; the `max` is what separates optimal from on-policy.
|
||||
4. `a*(s) = argmax_a Q*(s,a)` is why Q-learning is a viable family at all.
|
||||
5. Advantage `A = Q - V` is the relative-quality signal that policy gradients are built on.
|
||||
6. Log standard deviations are parameterized, not standard deviations, and state-independence
|
||||
is an implementation choice that differs across the algorithms in this book.
|
||||
|
||||
## Connects To
|
||||
- **Ch 8**: the taxonomy built on what each algorithm chooses to learn.
|
||||
- **Ch 9**: where the advantage function becomes the policy gradient weight.
|
||||
- **Ch 17**: the continuous-action argmax problem, straight out of `a*(s) = argmax_a Q*(s,a)`.
|
||||
- **Ch 12**: Exercise 1.1 asks you to implement the diagonal Gaussian log-likelihood above.
|
||||
|
|
@ -0,0 +1,123 @@
|
|||
# Chapter 8: Part 2 — Kinds of RL Algorithms
|
||||
|
||||
## Core Idea
|
||||
Two branching questions organize modern RL: does the agent have or learn a **model** of the
|
||||
environment, and **what does it learn** — a policy, a Q-function, a value function, or a
|
||||
model. Every trade-off in algorithm design descends from those two choices.
|
||||
|
||||
## Frameworks Introduced
|
||||
- **Branch 1: model-free vs model-based.** A model is a function predicting state transitions
|
||||
and rewards.
|
||||
- **Upside of a model: it allows the agent to plan** — think ahead, see what would happen
|
||||
across options, explicitly decide, then distil the planning results into a learned policy.
|
||||
AlphaZero is the famous case. When it works, it substantially improves sample efficiency.
|
||||
- **Downside: a ground-truth model is usually not available**, so it must be learned from
|
||||
experience, and **bias in a learned model gets exploited by the agent** — producing an
|
||||
agent that performs well against its own model and sub-optimally, or terribly, in the
|
||||
real environment. Model-learning is fundamentally hard; large amounts of time and compute
|
||||
can fail to pay off.
|
||||
- When to use model-based: you have a reliable model, or sample efficiency dominates and
|
||||
you can afford the model-bias risk.
|
||||
|
||||
- **Branch 2: what to learn in model-free RL — two families.**
|
||||
- **Policy Optimization.** Represent the policy explicitly as `pi_theta(a|s)` and optimize
|
||||
theta either directly by gradient ascent on `J(pi_theta)` or indirectly by maximizing
|
||||
local approximations of it. Almost always **on-policy**. Usually also learns an
|
||||
approximator `V_phi(s)` used in figuring out the policy update. Examples: A2C/A3C
|
||||
(direct), PPO (indirect, via a surrogate objective giving a conservative estimate of how
|
||||
much `J` will change).
|
||||
- **Q-Learning.** Learn an approximator `Q_theta(s,a)` to `Q*(s,a)`, usually with an
|
||||
objective based on the Bellman equation. Almost always **off-policy**. The policy comes
|
||||
from the connection `a(s) = argmax_a Q_theta(s,a)`. Examples: DQN, C51 (which learns a
|
||||
distribution over return whose expectation is `Q*`).
|
||||
|
||||
- **The policy-optimization vs Q-learning trade-off** (the single most quoted judgment in
|
||||
this book):
|
||||
- Policy optimization is **principled — you directly optimize for the thing you want** —
|
||||
which tends to make it stable and reliable.
|
||||
- Q-learning only *indirectly* optimizes agent performance, by training `Q_theta` to satisfy
|
||||
a self-consistency equation. **There are many failure modes for this kind of learning, so
|
||||
it tends to be less stable.**
|
||||
- But when Q-learning works it is **substantially more sample efficient**, because it can
|
||||
reuse data far more effectively.
|
||||
- How to use: pick the failure you can detect and afford. Instability you can see in a
|
||||
learning curve; sample inefficiency you can see in a compute bill.
|
||||
|
||||
- **The interpolation principle**: policy optimization and Q-learning are not incompatible,
|
||||
and under some circumstances turn out to be *equivalent* (Schulman et al 2017). A range of
|
||||
algorithms lives between them and trades off deliberately: **DDPG** (learns a deterministic
|
||||
policy and a Q-function that improve each other) and **SAC** (a variant using stochastic
|
||||
policies, entropy regularization and other tricks to stabilize learning; scores higher than
|
||||
DDPG on standard benchmarks).
|
||||
|
||||
- **Four ways to use a model** (there is no small set of clean clusters here; the model may
|
||||
be given or learned in each):
|
||||
1. **Pure planning** — never represent the policy at all. Model-predictive control (MPC):
|
||||
each time the agent observes, compute a plan optimal with respect to the model over a
|
||||
fixed window, execute only the first action, discard the rest, re-plan next step. Future
|
||||
rewards past the horizon can enter through a learned value function. Example: MBMF.
|
||||
2. **Expert iteration** — keep an explicit policy `pi_theta`, use a planning algorithm
|
||||
(e.g. Monte Carlo Tree Search) inside the model with candidate actions sampled from the
|
||||
current policy. The planner's output is an "expert" relative to the policy; update the
|
||||
policy toward it. Examples: ExIt, AlphaZero.
|
||||
3. **Data augmentation for model-free methods** — train a policy or Q-function with a
|
||||
model-free algorithm, but augment real experience with fictitious experience (MBVE), or
|
||||
train on purely fictitious experience ("training in the dream", World Models).
|
||||
4. **Embedding planning loops into policies** — make the planning procedure a subroutine of
|
||||
the policy so complete plans become side information, and train the policy output with
|
||||
any standard model-free algorithm. **The key advantage: model bias becomes less of a
|
||||
problem, because where the model is bad for planning the policy can learn to ignore it.**
|
||||
Example: I2A.
|
||||
|
||||
## Key Concepts
|
||||
- **Model of the environment**: a function predicting state transitions and rewards.
|
||||
- **Surrogate objective**: a local approximation of `J(pi_theta)` that is safe to maximize;
|
||||
the mechanism behind TRPO and PPO.
|
||||
- **The deadly triad**: function approximation + bootstrapping + off-policy data, which
|
||||
together cause instability in value-learning algorithms (Sutton and Barto ch. 11.3). This
|
||||
is the concrete content of "Q-learning has many failure modes."
|
||||
- **Modularity caveat**: the taxonomy is a tree and the real space is not. Advanced areas —
|
||||
exploration, transfer learning, meta learning — are omitted from it entirely.
|
||||
|
||||
## Reference Tables
|
||||
|
||||
| Choice | Buys you | Costs you |
|
||||
|--------|----------|-----------|
|
||||
| Model-based | Planning; large sample-efficiency gains when the model is good | Model must usually be learned; model bias gets exploited |
|
||||
| Model-free | Easier to implement and tune; more developed and tested | Forfeits the sample-efficiency gains a model could give |
|
||||
| Policy optimization | Stability, reliability, directly optimizes performance | Sample inefficiency (on-policy data only) |
|
||||
| Q-learning | Substantial sample efficiency through data reuse | Instability; no guarantee good Bellman fit means good policy |
|
||||
|
||||
## Mental Models
|
||||
- **Place any new algorithm with two questions**: model or no model, and what does it learn.
|
||||
That is what the taxonomy is for; it does not need to be exhaustive to do that job.
|
||||
- **Model bias is an adversarial problem, not a noise problem.** The agent is actively
|
||||
optimizing against your model's errors. That is why "the model is only 95% accurate" does
|
||||
not translate into "the policy is 95% as good."
|
||||
- **"Directly optimize the thing you want" is a design principle you can apply outside RL**:
|
||||
the further your training objective sits from your evaluation metric, the more failure modes
|
||||
you inherit.
|
||||
|
||||
## Anti-patterns
|
||||
- **Reading the taxonomy as exhaustive or as a strict tree.** The book opens with that
|
||||
disclaimer: algorithm modularity is not well represented by a tree.
|
||||
- **Assuming model-based is strictly better because it plans.** As of the guide's writing
|
||||
(September 2018), model-free methods were more popular and more extensively developed and
|
||||
tested, precisely because model-learning is hard.
|
||||
- **Treating "it fits the Bellman equation well" as evidence the policy is good.** No such
|
||||
guarantee exists — this is the defining weakness of the whole Q-learning family.
|
||||
|
||||
## Key Takeaways
|
||||
1. Two branching questions — model or not, and what to learn — generate the whole landscape.
|
||||
2. Policy optimization: principled, stable, sample-hungry. Q-learning: sample-efficient,
|
||||
indirect, unstable.
|
||||
3. The two families are not exclusive; DDPG and SAC live between them on purpose.
|
||||
4. Model bias is exploited by the agent, which is the central risk of model-based methods.
|
||||
5. Embedding a planner inside a policy is the model-based approach that most directly
|
||||
defuses model bias, because the policy can learn when to ignore the model.
|
||||
|
||||
## Connects To
|
||||
- **Ch 3**: the six implemented algorithms, which are all model-free.
|
||||
- **Ch 9**: the mathematics of the policy-optimization family.
|
||||
- **Ch 17 and Ch 19**: DDPG and SAC as the concrete interpolations named here.
|
||||
- **Ch 11**: the key-papers list, whose top-level sections mirror this taxonomy.
|
||||
|
|
@ -0,0 +1,161 @@
|
|||
# Chapter 9: Part 3 — Intro to Policy Optimization
|
||||
|
||||
## Core Idea
|
||||
Three results build the policy gradient you actually implement: the simplest analytical
|
||||
expression for `grad J`, a rule that lets you **drop useless terms** (reward-to-go), and a
|
||||
rule that lets you **add useful terms** (baselines) — ending at the advantage-weighted form
|
||||
`grad J = E[ sum_t grad log pi_theta(a_t|s_t) * A^pi(s_t,a_t) ]`.
|
||||
|
||||
## Frameworks Introduced
|
||||
- **The general policy gradient form**: every variant is
|
||||
`grad_theta J(pi_theta) = E_{tau~pi_theta}[ sum_{t=0}^{T} grad_theta log pi_theta(a_t|s_t) * Phi_t ]`
|
||||
and the whole subject is *which* `Phi_t` you choose. Five valid choices, all with the same
|
||||
expectation and different variance:
|
||||
1. `Phi_t = R(tau)` — the full trajectory return (the simplest form)
|
||||
2. `Phi_t = sum_{t'=t}^{T} R(s_t', a_t', s_{t'+1})` — the **reward-to-go**
|
||||
3. `Phi_t = reward-to-go - b(s_t)` — reward-to-go with a **baseline**
|
||||
4. `Phi_t = Q^{pi_theta}(s_t, a_t)`
|
||||
5. `Phi_t = A^{pi_theta}(s_t, a_t)` — the **advantage**, the one Spinning Up's VPG uses
|
||||
- When to use: choice 5 in practice; the list is the map for reading any policy gradient paper.
|
||||
|
||||
- **The derivation, in five facts.** Worth memorizing because every extension reuses them:
|
||||
1. `P(tau|theta) = rho_0(s_0) * prod_t P(s_{t+1}|s_t,a_t) * pi_theta(a_t|s_t)`
|
||||
2. **Log-derivative trick**: `grad_theta P(tau|theta) = P(tau|theta) * grad_theta log P(tau|theta)`
|
||||
3. `log P(tau|theta) = log rho_0(s_0) + sum_t [ log P(s_{t+1}|s_t,a_t) + log pi_theta(a_t|s_t) ]`
|
||||
4. **Gradients of environment functions are zero** — the environment has no dependence on theta,
|
||||
so `rho_0`, `P` and `R` all vanish under `grad_theta`.
|
||||
5. Therefore `grad_theta log P(tau|theta) = sum_t grad_theta log pi_theta(a_t|s_t)`.
|
||||
- Chained: expand the expectation, bring the gradient inside the integral, apply the
|
||||
log-derivative trick, return to expectation form, substitute fact 5.
|
||||
|
||||
- **The EGLP lemma (Expected Grad-Log-Prob)**: for any parameterized distribution `P_theta`,
|
||||
`E_{x~P_theta}[ grad_theta log P_theta(x) ] = 0`.
|
||||
- Proof in three lines: all distributions are normalized (`integral P_theta(x) = 1`); take
|
||||
the gradient of both sides (`= grad 1 = 0`); apply the log-derivative trick.
|
||||
- When to use: it is the engine behind both the drop rule and the add rule. The author notes
|
||||
it has no standard name in the literature but comes up often enough to deserve one.
|
||||
|
||||
- **"Don't let the past distract you" (the drop rule)**: agents should only reinforce actions
|
||||
on the basis of their **consequences**. Rewards obtained *before* an action have no bearing
|
||||
on how good that action was. Formally, all terms with `t' < t` are zero in expectation.
|
||||
- Why it is better, precisely: those dropped terms had **zero mean but nonzero variance** —
|
||||
they added pure noise to the sample estimate. Removing them reduces the number of sample
|
||||
trajectories needed.
|
||||
|
||||
- **Baselines (the add rule)**: an immediate consequence of EGLP is that for any function `b`
|
||||
depending only on state, `E_{a_t~pi}[ grad log pi(a_t|s_t) * b(s_t) ] = 0`. So you may add
|
||||
or subtract any such term without changing the gradient in expectation.
|
||||
- **The most common baseline is the on-policy value function `V^pi(s_t)`**, which empirically
|
||||
reduces variance and gives faster, more stable learning.
|
||||
- The conceptual appeal: it encodes the intuition that **if an agent gets what it expected,
|
||||
it should "feel" neutral about it.**
|
||||
- In practice `V^pi` cannot be computed exactly, so it is approximated by a network `V_phi`
|
||||
updated concurrently with the policy (so it always approximates the *most recent* policy).
|
||||
The simplest learning rule, used by VPG, TRPO, PPO and A2C, is mean-squared error:
|
||||
`phi_k = argmin_phi E_{s_t, Rhat_t ~ pi_k}[ (V_phi(s_t) - Rhat_t)^2 ]`, via one or more
|
||||
gradient steps starting from `phi_{k-1}`.
|
||||
|
||||
## Key Concepts
|
||||
- **Policy gradient**: `grad_theta J(pi_theta)`. Algorithms that optimize this way are
|
||||
policy gradient algorithms — VPG and TRPO are; **PPO is often called one though this is
|
||||
slightly inaccurate.**
|
||||
- **Reward-to-go**: `Rhat_t = sum_{t'=t}^{T} R(s_t', a_t', s_{t'+1})`.
|
||||
- **Baseline**: any state-only function subtracted from the weight.
|
||||
- **Sample estimate**: `ghat = (1/|D|) * sum_{tau in D} sum_t grad log pi_theta(a_t|s_t) * Phi_t`.
|
||||
- **GAE (Generalized Advantage Estimation)**: the widely-used method for approximating the
|
||||
advantage function; Spinning Up's VPG, TRPO and PPO all use it. The book strongly advises
|
||||
studying the paper.
|
||||
- **Epoch** (in this context): one experience-collection phase plus one policy gradient update.
|
||||
|
||||
## Code Examples
|
||||
The whole simple algorithm is 128 lines (`spinup/examples/pytorch/pg_math/1_simple_pg.py`).
|
||||
The three pieces that matter:
|
||||
|
||||
```python
|
||||
# 1. Policy network
|
||||
logits_net = mlp(sizes=[obs_dim] + hidden_sizes + [n_acts])
|
||||
|
||||
def get_policy(obs):
|
||||
return Categorical(logits=logits_net(obs))
|
||||
|
||||
def get_action(obs):
|
||||
return get_policy(obs).sample().item()
|
||||
|
||||
# 2. The "loss" whose gradient is the policy gradient
|
||||
def compute_loss(obs, act, weights):
|
||||
logp = get_policy(obs).log_prob(act)
|
||||
return -(logp * weights).mean()
|
||||
|
||||
# 3. One gradient step
|
||||
optimizer.zero_grad()
|
||||
batch_loss = compute_loss(obs=..., act=..., weights=...)
|
||||
batch_loss.backward()
|
||||
optimizer.step()
|
||||
```
|
||||
|
||||
Upgrading to reward-to-go changes one function and two lines:
|
||||
|
||||
```python
|
||||
def reward_to_go(rews):
|
||||
n = len(rews)
|
||||
rtgs = np.zeros_like(rews)
|
||||
for i in reversed(range(n)):
|
||||
rtgs[i] = rews[i] + (rtgs[i+1] if i+1 < n else 0)
|
||||
return rtgs
|
||||
|
||||
# was: batch_weights += [ep_ret] * ep_len
|
||||
batch_weights += list(reward_to_go(ep_rews))
|
||||
```
|
||||
|
||||
For a diagonal Gaussian policy, `log_prob(act)` returns per-component log probabilities of
|
||||
shape `(batch, act_dim)` when RL needs shape `(batch,)`. Sum them:
|
||||
|
||||
```python
|
||||
logp = get_policy(obs).log_prob(act).sum(axis=-1)
|
||||
```
|
||||
|
||||
## Worked Example
|
||||
**The policy gradient "loss" is not a loss function.** This is the single most consequential
|
||||
warning in the chapter, and it differs from supervised learning in two ways:
|
||||
|
||||
1. **The data distribution depends on the parameters.** A supervised loss is defined on a
|
||||
fixed data distribution independent of the parameters being optimized. Here the data must
|
||||
be sampled from the most recent policy.
|
||||
2. **It does not measure performance.** We care about `J(pi_theta)`, and this "loss" does not
|
||||
approximate it — not even in expectation. It is useful only because, *evaluated at the
|
||||
current parameters with data generated by those parameters*, it has the negative gradient
|
||||
of performance.
|
||||
|
||||
After the first gradient step there is no connection to performance at all. You can send
|
||||
this loss to negative infinity while policy performance craters — and it usually will.
|
||||
Researchers sometimes call this the policy "overfitting" to a batch; the phrase is
|
||||
descriptive but should not be taken literally, since it does not refer to generalization error.
|
||||
|
||||
**The practical rule: in policy gradients, only average return means anything. The loss
|
||||
function means nothing.** The ML habit of reading "loss went down, all is well" is wrong here.
|
||||
|
||||
## Anti-patterns
|
||||
- **Using the loss curve as a training health signal.** See above. Watch `AverageEpRet`.
|
||||
- **Weighting every action by the full-trajectory return** when reward-to-go is a two-line
|
||||
change that strictly reduces variance.
|
||||
- **Forgetting to sum log-probabilities across action dimensions** for Gaussian policies —
|
||||
a shape bug that will not raise, in the same family as the ch12 DDPG bug.
|
||||
- **Treating a baseline as a bias**: baselines are provably zero-mean under EGLP. They change
|
||||
variance, never the expected gradient.
|
||||
|
||||
## Key Takeaways
|
||||
1. One template, five valid weights `Phi_t`; the advantage form is the destination.
|
||||
2. The whole derivation rests on the log-derivative trick plus the fact that environment
|
||||
gradients vanish.
|
||||
3. EGLP is a single lemma that licenses both dropping past rewards and adding baselines.
|
||||
4. Reward-to-go removes terms with zero mean and nonzero variance — pure noise reduction.
|
||||
5. The value baseline is learned by mean-squared regression onto reward-to-go, concurrently
|
||||
with the policy.
|
||||
6. The policy gradient loss is not a performance measure and must never be read as one.
|
||||
|
||||
## Connects To
|
||||
- **Ch 7**: the advantage function and value functions this chapter puts to work.
|
||||
- **Ch 14**: VPG, the direct implementation of the final advantage-weighted form.
|
||||
- **Ch 15 and Ch 16**: TRPO and PPO, which replace the plain gradient step with a
|
||||
trust-region-constrained one.
|
||||
- **Ch 12**: Exercise 2.1 measures exactly what happens when `V_phi` is not fit properly.
|
||||
|
|
@ -0,0 +1,133 @@
|
|||
# Chapter 10: Spinning Up as a Deep RL Researcher
|
||||
|
||||
## Core Idea
|
||||
Joshua Achiam's curriculum for becoming a deep RL researcher, in four stages: build the
|
||||
right background, learn by writing your own implementations, develop a research project
|
||||
through one of three idea frames, and hold yourself to experimental standards strict enough
|
||||
that a positive result means something.
|
||||
|
||||
## Frameworks Introduced
|
||||
- **The Right Background** (four items, all prerequisites, none optional):
|
||||
- **Math**: from probability and statistics — random variables, Bayes' theorem, chain rule
|
||||
of probability, expected values, standard deviations, importance sampling. From
|
||||
multivariate calculus — gradients, and optionally Taylor series expansions.
|
||||
- **General deep learning**: standard architectures (MLP, vanilla RNN, LSTM, GRU, conv
|
||||
layers, resnets, attention), regularizers (weight decay, dropout), normalization (batch,
|
||||
layer, weight norm), optimizers (SGD, momentum SGD, Adam), and the reparameterization trick.
|
||||
- **One deep learning library**, well enough to confidently implement a simple supervised
|
||||
learning program.
|
||||
- **RL concepts and terminology**: states, actions, trajectories, policies, rewards, value
|
||||
functions, action-value functions. Optionally, monotonic improvement theory (the basis
|
||||
for advanced policy gradient algorithms) and classical RL algorithms.
|
||||
|
||||
- **Learn by Doing** — the core of the essay, in eight rules:
|
||||
1. **Write your own implementations**, aiming for the **shortest correct implementation** of
|
||||
each. By far the best way to build both understanding and performance intuitions.
|
||||
2. **Simplicity is critical.** Implement the simplest algorithms first and add complexity
|
||||
gradually. Starting with too many moving parts means weeks lost to debugging — a common
|
||||
failure mode for newcomers. If you are stuck in it, drop to a simpler algorithm and
|
||||
return later.
|
||||
3. **Which algorithms, in roughly this order**: VPG (REINFORCE), DQN, A2C (the synchronous
|
||||
A3C), PPO (clipped objective), DDPG. The simplest versions are a few hundred lines
|
||||
(ballpark 250-300); a no-frills VPG is about 80. Write single-threaded before parallel,
|
||||
but do parallelize at least one.
|
||||
4. **Focus on understanding, because broken RL code almost always fails silently** — the
|
||||
code runs fine and the agent simply never learns. Usually something is computed with the
|
||||
wrong equation, on the wrong distribution, or piped to the wrong place. Sometimes the
|
||||
only way to find it is to read the code critically, knowing exactly what it should do.
|
||||
5. **What to look for in papers**: scour ablation analyses (they tell you which parameters
|
||||
and subroutines actually matter, which is what you need to diagnose bugs) and
|
||||
supplementary material (network architectures, optimization hyperparameters — align your
|
||||
implementation to these to improve your odds of getting it working).
|
||||
6. **But do not overfit to paper details.** Papers often prescribe more tricks than are
|
||||
strictly necessary. The original DDPG paper suggests a complex architecture, an
|
||||
initialization scheme and batch normalization; none are strictly necessary and some of
|
||||
the best DDPG results use simpler networks. The original A3C uses asynchronous updates
|
||||
from actor-learners; synchronous updates work about as well.
|
||||
7. **Do not overfit to existing implementations either.** RL libraries make abstraction
|
||||
choices that are good for code reuse across algorithms but unnecessary if you are
|
||||
writing one algorithm for one use case.
|
||||
8. **Iterate fast in simple environments.** CartPole-v0, InvertedPendulum-v0, FrozenLake-v0,
|
||||
HalfCheetah-v2 with a short horizon (100 or 250 steps, not the full 1000).
|
||||
**Ideal debug-stage turnaround: under 5 minutes on your local machine.** These runs need
|
||||
no special hardware and run fine on CPUs. Do not attempt Atari or Humanoid before the
|
||||
simplest toy task works.
|
||||
|
||||
- **Three idea-generation frames** — the frame you pick shapes the project's scope and risk:
|
||||
|
||||
| Frame | What it is | Scope | Main risk |
|
||||
|-------|-----------|-------|-----------|
|
||||
| **1. Improving on an existing approach** | Incrementalism: performance gains in an established setting by tweaking an existing algorithm. Reimplementing prior work exposes where it is brittle. Most accessible to novices, still worthwhile at any level | Narrow, wraps up in a few months | Your tweaks may simply fail to improve it, and then the project is over with no signal on what to do next |
|
||||
| **2. Focusing on unsolved benchmarks** | Succeed at a task nobody has solved. You may try a wide variety of methods, prior and invented | Broad, several months to a year-plus | The benchmark may be unsolvable without a substantial breakthrough; easy to spend a long time with no progress. Even failure usually yields insights that seed the next project |
|
||||
| **3. Creating a new problem setting** | An entirely different conceptual problem nobody has studied; you will have to design the benchmark too | Open-ended | Enormous challenge — but great benchmarks move the whole field forward. These problems come up when they come up; hard to go looking for them |
|
||||
|
||||
- **Doing Rigorous Research in RL** — four standards, each closing a specific way a result
|
||||
can look real and not be:
|
||||
1. **Set up fair comparisons.** If you implement your baseline from scratch, spend **as much
|
||||
time tuning the baseline as tuning your own algorithm.** Hold all else equal even when
|
||||
the methods differ substantially — e.g. keep parameter counts approximately equal when
|
||||
investigating architecture variants. **Under no circumstances handicap the baseline.**
|
||||
RL baselines are strong, and consistent wins over them are hard.
|
||||
2. **Remove stochasticity as a confounder.** Run everything for many random seeds — **at
|
||||
least 3, and 10 or more to be thorough.** Deep RL is fairly brittle with respect to seed;
|
||||
two different groups of seeds can produce learning curves so different they look like
|
||||
they come from different distributions.
|
||||
3. **Run high-integrity experiments.** Do not report the best or most interesting runs.
|
||||
Launch **new, final experiments** for every method being compared and **precommit to
|
||||
reporting whatever comes out.** This enforces a weak form of preregistration: the tuning
|
||||
stage produces your hypotheses, the final runs produce your conclusions.
|
||||
4. **Check each claim separately — run an ablation analysis.** Any proposed method has
|
||||
several key design decisions, and the claim "these collectively help" is really a bundle
|
||||
of separate claims. Systematically swap or remove each one to attribute credit correctly.
|
||||
This lets you state each claim with a measure of confidence and strengthens the whole work.
|
||||
|
||||
## Key Concepts
|
||||
- **Silent failure**: the defining property of broken RL code. It runs; the agent just never learns.
|
||||
- **Measure everything**: instrument heavily. The author's own list — mean/std/min/max of
|
||||
cumulative rewards, episode lengths and value function estimates, plus the objective losses
|
||||
and any exploration parameters (mean policy entropy for stochastic policy optimization,
|
||||
current epsilon for epsilon-greedy). **Also watch videos of your agent** periodically; it
|
||||
gives insights nothing else does. You cannot tell it is broken if you cannot see it breaking.
|
||||
- **Assume there is a bug.** Spend a lot of effort searching for bugs before tweaking
|
||||
hyperparameters. Bad hyperparameters can significantly degrade performance, but if yours
|
||||
are similar to those in papers and standard implementations, they are probably not the issue.
|
||||
- **Test in more than one environment**: sometimes code works in one environment despite a
|
||||
breaking bug, so re-verify once results look promising.
|
||||
- **Avoid reinventing the wheel**: before investing, check thoroughly that the idea has not
|
||||
been done. But do not let the risk push you into planting flags with not-quite-finished
|
||||
research or over-claiming partial work. Complete, thorough investigations are what counts.
|
||||
- **Scale when things work**: after the simplest environments pass, move to harder ones —
|
||||
experiments now take hours to a couple of days, and specialized hardware (a beefy GPU, a
|
||||
32-core machine) or cloud resources start to be worth it.
|
||||
|
||||
## Anti-patterns
|
||||
- **Tuning hyperparameters to fix a bug.** It is usually a bug.
|
||||
- **Debugging in a hard environment.** If turnaround exceeds a few minutes at the debug stage,
|
||||
you are working in the wrong environment.
|
||||
- **Starting with the complex algorithm.** The predicted outcome is weeks lost.
|
||||
- **Reporting your best run.** That is the failure the precommitment rule exists to stop.
|
||||
- **Under-tuning the baseline** — the most common way to manufacture a positive result without
|
||||
intending to.
|
||||
- **Fewer than 3 seeds.** With deep RL's seed sensitivity, a single-seed result carries
|
||||
almost no information.
|
||||
- **Bundling claims.** Without ablations you cannot say which of your design decisions did
|
||||
the work — including, possibly, none of them.
|
||||
|
||||
## Key Takeaways
|
||||
1. Implement from scratch, simplest first, shortest correct version, single-threaded before parallel.
|
||||
2. Broken RL code fails silently — so instrumentation and critical reading, not error messages,
|
||||
are your debugging tools.
|
||||
3. Debug loops under five minutes in toy environments; scale only after correctness.
|
||||
4. Read papers for ablations and supplementary material, but do not adopt every trick.
|
||||
5. Pick an idea frame deliberately: incremental (narrow, fast, may dead-end), unsolved
|
||||
benchmark (broad, slow, insight-rich even in failure), or new problem setting (rare, huge).
|
||||
6. Rigor is four things: a fully-tuned baseline, many seeds, precommitted final runs, and
|
||||
per-claim ablations.
|
||||
7. These habits are worth keeping past the learning stage; they accelerate research.
|
||||
|
||||
## Connects To
|
||||
- **Ch 1**: the code design philosophy that makes the reference implementations readable.
|
||||
- **Ch 4**: the `--seed 0 10 20` flag that operationalizes the multi-seed rule.
|
||||
- **Ch 11**: the key papers list, the recommended starting point for literature exploration.
|
||||
- **Ch 12**: the exercises — Problem Set 2 is entirely about silent failure modes.
|
||||
- **Ch 13**: the benchmarks, run at 10 seeds — the standard this chapter asks for.
|
||||
|
|
@ -0,0 +1,70 @@
|
|||
# Chapter 11: Key Papers in Deep RL
|
||||
|
||||
## Core Idea
|
||||
A curated, explicitly non-comprehensive reading list of roughly 100 papers organized into
|
||||
13 topic areas — designed as a starting point for someone looking to do research, and as
|
||||
the map you use to pick a research topic in ch10's "explore the literature" step.
|
||||
|
||||
## Frameworks Introduced
|
||||
- **Use the list as a topic map, not a queue.** The intended workflow (from ch10): scan the
|
||||
sections to become aware of what topics exist, find a paper on one that inspires you, read
|
||||
it thoroughly, then use its related-work section and citations to do a deep dive. You will
|
||||
start to see where the unsolved problems are.
|
||||
- **Each entry is annotated with what it contributes** — either **Algorithm: X** (this paper
|
||||
introduced X) or **Contribution: ...** (this paper established a result, a critique or a
|
||||
codebase). Scan the annotations, not the titles.
|
||||
|
||||
## Reference Tables
|
||||
|
||||
The 13 sections, with the subsections that reveal the field's internal structure:
|
||||
|
||||
| # | Section | Subsections / notable entries |
|
||||
|---|---------|------------------------------|
|
||||
| 1 | **Model-Free RL** | a. Deep Q-Learning (DQN, Deep Recurrent Q-Learning, Dueling DQN, Double DQN, Prioritized Experience Replay, Rainbow) · b. Policy Gradients (A3C, TRPO, GAE, PPO, ACKTR, ACER, SAC) · c. Deterministic Policy Gradients (DPG, DDPG, TD3) · d. Distributional RL (C51, QR-DQN, IQN, Dopamine) · e. Policy Gradients with Action-Dependent Baselines (Q-Prop, Stein Control Variates, and Tucker et al's critique) · f. Path-Consistency Learning (PCL, Trust-PCL) · g. Other Ways of Combining Policy-Learning and Q-Learning (PGQL, Reactor, IPG, the policy-gradient/soft-Q-learning equivalence) · h. Evolutionary Algorithms (ES) |
|
||||
| 2 | **Exploration** | a. Intrinsic Motivation (VIME, count-based pseudocounts, hash-based counts, EX2, ICM, RND) · b. Unsupervised RL (VIC, DIAYN, VALOR) |
|
||||
| 3 | **Transfer and Multitask RL** | Progressive Networks, UVFA, UNREAL, PathNet, MATL, HER |
|
||||
| 4 | **Hierarchy** | STRAW, Feudal Networks, HIRO |
|
||||
| 5 | **Memory** | MFEC, NEC, Neural Map, MERLIN, RMC |
|
||||
| 6 | **Model-Based RL** | a. Model is Learned (I2A, MBMF, MVE) · b. Model is Given (AlphaZero and relatives) |
|
||||
| 7 | **Meta-RL** | |
|
||||
| 8 | **Scaling RL** | |
|
||||
| 9 | **RL in the Real World** | |
|
||||
| 10 | **Safety** | |
|
||||
| 11 | **Imitation Learning and Inverse Reinforcement Learning** | |
|
||||
| 12 | **Reproducibility, Analysis, and Critique** | |
|
||||
| 13 | **Bonus: Classic Papers in RL Theory or Review** | |
|
||||
|
||||
## Mental Models
|
||||
- **Section 1's shape mirrors ch8's taxonomy**: Q-learning, policy gradients, and the
|
||||
deterministic and interpolating methods between them. If you understood ch8, section 1
|
||||
is already organized in your head.
|
||||
- **Section 12 is the one people skip and shouldn't.** "Reproducibility, Analysis, and
|
||||
Critique" is where ch10's rigor standards come from, and where you learn that published
|
||||
gains sometimes do not survive re-examination — the Tucker et al entry in section 1e is a
|
||||
worked example: it critiques and re-evaluates claims from earlier papers (including Q-Prop
|
||||
and Stein control variates) and finds important methodological errors in them.
|
||||
- **A reading list is a topic-awareness tool.** The list explicitly does not claim
|
||||
completeness; its job is to prevent you from picking a research problem while unaware that
|
||||
a whole subfield already works on it.
|
||||
|
||||
## Anti-patterns
|
||||
- **Reading it front to back.** It is far from comprehensive and is not a curriculum; the
|
||||
intended use is targeted depth after topic selection.
|
||||
- **Treating an entry as endorsement of the result.** Several entries exist precisely because
|
||||
they critique other entries.
|
||||
- **Skipping the related-work walk.** The list is the entry point; the citations are the
|
||||
actual literature review.
|
||||
|
||||
## Key Takeaways
|
||||
1. 13 topic sections; section 1 (Model-Free RL) alone has eight subsections and mirrors ch8.
|
||||
2. Every entry is annotated with the algorithm it introduced or the contribution it made.
|
||||
3. The list is a topic map for choosing a research direction, not a reading queue.
|
||||
4. Sections 7-13 (meta-RL, scaling, real world, safety, imitation/IRL, critique, classics)
|
||||
are where the topics ch10 suggests for project ideas actually live.
|
||||
|
||||
## Connects To
|
||||
- **Ch 8**: the taxonomy that section 1's structure reproduces.
|
||||
- **Ch 10**: "start by exploring the literature to become aware of topics in the field" —
|
||||
this chapter is that step's tool.
|
||||
- **Ch 14-19**: every implemented algorithm's own "Why These Papers?" section is a
|
||||
three-paper version of this list, scoped to one algorithm.
|
||||
|
|
@ -0,0 +1,105 @@
|
|||
# Chapter 12: Exercises
|
||||
|
||||
## Core Idea
|
||||
Two problem sets with opposite purposes: Problem Set 1 makes you write the mathematical
|
||||
core of three algorithms, and Problem Set 2 makes you **watch RL code fail silently** and
|
||||
find out why — because that is the skill the essay says decides whether you can do this work.
|
||||
|
||||
## Frameworks Introduced
|
||||
- **Problem Set 1: Basics of Implementation** — you are given everything except the math.
|
||||
- **1.1 Gaussian Log-Likelihood.** Write a function taking means, log stds and samples,
|
||||
returning the log likelihoods. Auto-checked against a known-good implementation on a
|
||||
batch of random inputs. The formula is the one in ch7.
|
||||
- **1.2 Policy for PPO.** Implement an MLP diagonal Gaussian policy. Auto-checked by
|
||||
running 20 epochs on InvertedPendulum-v2 (3-5 minutes). **Bar for success: average score
|
||||
over 500 in the last 5 epochs, or 1000 (the maximum) in the last 5.**
|
||||
- **1.3 Computation Graph for TD3.** You are given the entire TD3 algorithm *except* the
|
||||
loss functions and the intermediate calculations for them. Find "YOUR CODE HERE".
|
||||
No automatic checking. Evaluated on HalfCheetah-v2, InvertedPendulum-v2 and one other
|
||||
MuJoCo environment of your choosing, with smaller networks ([128,128]), max episode
|
||||
length 150, 10 epochs, roughly 10 minutes. **Anecdotal targets within 10 epochs:
|
||||
HalfCheetah over 300, InvertedPendulum maxing out at 150.** `--use_soln` runs Spinning
|
||||
Up's TD3 instead of yours.
|
||||
|
||||
- **Problem Set 2: Algorithm Failure Modes** — the point is the failure, not the fix.
|
||||
- **2.1 Value Function Fitting in TRPO.** Compare `train_v_iters=80` against
|
||||
`train_v_iters=0` on Hopper-v2, three seeds each, 250 epochs, 4000 steps per epoch.
|
||||
**Result: the difference is substantial. With a trained value function the agent makes
|
||||
quick progress; with an untrained one it gets stuck early on.** Few factors affect policy
|
||||
gradient performance more drastically than the quality of the value function used for
|
||||
advantage estimation.
|
||||
- **2.2 Silent Bug in DDPG.** Run DDPG with and without a planted bug, three seeds each,
|
||||
six runs, ~10 minutes each; plot and compare. Then, **without looking at DDPG's `core.py`**,
|
||||
work out what the bug is.
|
||||
|
||||
## Worked Example
|
||||
**The 2.2 bug, and why it is the most instructive page in the book.**
|
||||
|
||||
The correct and bugged actor-critic differ in exactly one thing: whether the Q-function
|
||||
output is squeezed.
|
||||
|
||||
```python
|
||||
# Correct
|
||||
def forward(self, obs, act):
|
||||
q = self.q(torch.cat([obs, act], dim=-1))
|
||||
return torch.squeeze(q, -1) # Critical to ensure q has right shape.
|
||||
|
||||
# Bugged
|
||||
def forward(self, obs, act):
|
||||
return self.q(torch.cat([obs, act], dim=-1)) # shape [batch, 1], not [batch]
|
||||
```
|
||||
|
||||
The TF1 version is the same defect: the correct code squeezes to shape `[batch size]`, the
|
||||
bugged code leaves shape `[batch size, 1]`.
|
||||
|
||||
Why that one missing squeeze destroys learning — look at the DDPG graph:
|
||||
|
||||
```python
|
||||
backup = tf.stop_gradient(r_ph + gamma*(1-d_ph)*q_pi_targ)
|
||||
pi_loss = -tf.reduce_mean(q_pi)
|
||||
q_loss = tf.reduce_mean((q - backup)**2)
|
||||
```
|
||||
|
||||
`r_ph` and `d_ph` have shape `[batch size]`. The backup line was written assuming it adds
|
||||
tensors of the same shape. But it will also happily add tensors of *different* shapes as long
|
||||
as they are broadcast-compatible — and `[batch size]` and `[batch size, 1]` are compatible,
|
||||
with results that are not what you expect:
|
||||
|
||||
```
|
||||
x has shape [5], y has shape [5,1]
|
||||
x * y -> shape [5,5]
|
||||
x + y -> shape [5,5]
|
||||
```
|
||||
|
||||
Adding or multiplying a shape-`[5]` tensor by a shape-`[5,1]` tensor returns a shape-`[5,5]`
|
||||
tensor. So when the Q-functions are not squeezed, `q_pi_targ` has shape `[batch size, 1]`,
|
||||
the backup becomes a `[batch, batch]` matrix, and the whole Q-loss is meaningless. **Nothing
|
||||
raises. Nothing warns. The run completes. The agent just learns worse.**
|
||||
|
||||
This is ch10's "broken RL code almost always fails silently" reduced to a single missing
|
||||
`squeeze`. The **Bonus** question is worth sitting with: *are there any choices of
|
||||
hyperparameters which would have hidden the effects of the bug?*
|
||||
|
||||
## Anti-patterns
|
||||
- **Looking at `core.py` before attempting 2.2.** The exercise is diagnostic practice; the
|
||||
answer is worth less than the search.
|
||||
- **Broadcast-compatible shapes as an implicit contract.** The lesson generalizes far past
|
||||
DDPG: any elementwise op between a `[N]` and an `[N,1]` tensor is a silent bug waiting.
|
||||
Assert shapes.
|
||||
- **Skipping Problem Set 2 because it produces no code.** 2.1 produces no code at all and is
|
||||
still one of the most decision-relevant results in the book.
|
||||
|
||||
## Key Takeaways
|
||||
1. Problem Set 1 is math-in-code: Gaussian log-likelihood, a PPO policy, TD3's losses.
|
||||
2. 1.1 and 1.2 self-check; 1.3 does not, and is evaluated on visible learning progress.
|
||||
3. 2.1: a badly-fit value function does not degrade a policy gradient agent gracefully — it
|
||||
gets stuck early.
|
||||
4. 2.2: one missing `squeeze` silently corrupts the Bellman backup through broadcasting.
|
||||
5. The challenges past the problem sets are: reimplement algorithms from scratch (ch10), and
|
||||
attempt OpenAI's standing Requests for Research.
|
||||
|
||||
## Connects To
|
||||
- **Ch 7**: the diagonal Gaussian log-likelihood formula that Exercise 1.1 asks for.
|
||||
- **Ch 9**: why the value function baseline quality drives policy gradient performance (2.1).
|
||||
- **Ch 17 and Ch 18**: the DDPG graph and the TD3 losses the exercises operate on.
|
||||
- **Ch 10**: "broken RL code almost always fails silently" — Problem Set 2 is the proof.
|
||||
|
|
@ -0,0 +1,87 @@
|
|||
# Chapter 13: Benchmarks for Spinning Up Implementations
|
||||
|
||||
## Core Idea
|
||||
All six algorithms were benchmarked for 3M timesteps across five MuJoCo environments at
|
||||
10 seeds each — and the results carry an explicit, unusual disclosure about which
|
||||
implementations are research-grade and which are not.
|
||||
|
||||
## Frameworks Introduced
|
||||
- **The parity disclosure — the single most decision-relevant paragraph in the chapter:**
|
||||
- **DDPG, TD3 and SAC are roughly at parity with the best reported results** for those
|
||||
algorithms. **You can use them for research purposes.**
|
||||
- **VPG, TRPO and PPO are overall a bit weaker than the best reported results**, because
|
||||
the implementations omit standard tricks — specifically **observation normalization** and
|
||||
**normalized value regression targets**. **For research comparisons, use the TRPO or PPO
|
||||
implementations from OpenAI Baselines instead.**
|
||||
- When to use: before citing any Spinning Up number, or before using one of these as a
|
||||
baseline in a paper (which ch10 says you must tune as hard as your own method).
|
||||
|
||||
- **Report performance with its measurement definition attached.** The two families are not
|
||||
measured the same way, and saying "return" without saying which is a category error:
|
||||
- **On-policy**: the average trajectory return across the batch collected at each epoch —
|
||||
i.e. the exploring policy, measured continuously.
|
||||
- **Off-policy**: measured **once every 10,000 steps** by running the deterministic policy
|
||||
(or, for SAC, the **mean** policy) **without action noise** for **ten trajectories**, and
|
||||
reporting the average return over those test trajectories.
|
||||
|
||||
## Reference Tables
|
||||
|
||||
Benchmark setup:
|
||||
|
||||
| Parameter | On-policy (VPG, TRPO, PPO) | Off-policy (DDPG, TD3, SAC) |
|
||||
|-----------|---------------------------|----------------------------|
|
||||
| Network architecture | (64, 32), tanh units | (256, 256), relu units |
|
||||
| Batch | 4000 environment steps per batch update | Minibatches of 100 per gradient step |
|
||||
| Performance metric | Average batch trajectory return per epoch | Deterministic/mean policy, no noise, 10 trajectories every 10k steps |
|
||||
| Research-grade? | No — use Baselines for TRPO/PPO | Yes |
|
||||
|
||||
Common to all:
|
||||
|
||||
| Setting | Value |
|
||||
|---------|-------|
|
||||
| Environments | HalfCheetah-v3, Hopper-v3, Walker2d-v3, Swimmer-v3, Ant-v3 (MuJoCo Gym suite) |
|
||||
| Timesteps | 3M |
|
||||
| Random seeds | 10 per experiment |
|
||||
| Plot content | Solid line = mean over seeds; shaded = std dev over seeds |
|
||||
| Smoothing | Averaged over a window of 11 epochs |
|
||||
| Other hyperparameters | Left at Spinning Up defaults; see each algorithm page |
|
||||
|
||||
Both PyTorch and TF1 versions were benchmarked in every environment, plus dedicated
|
||||
head-to-head PyTorch-vs-TF1 pages for VPG, PPO, DDPG, TD3 and SAC. (TRPO has no PyTorch
|
||||
implementation, so it has no head-to-head page.)
|
||||
|
||||
## Mental Models
|
||||
- **Reporting mean and std dev over 10 seeds *as shaded bands* is the visual form of ch10's
|
||||
"remove stochasticity as a confounder."** A benchmark plot without a seed band is telling
|
||||
you less than it appears to.
|
||||
- **The architecture split is itself a finding**: on-policy at (64, 32)/tanh and off-policy
|
||||
at (256, 256)/relu is the convention these algorithm families settled into, not an
|
||||
arbitrary choice, and copying the wrong family's defaults is a real source of bad results.
|
||||
- **Honest self-assessment is a feature of the resource**, not a caveat. A benchmarks page
|
||||
that names which of its own implementations you should not use is doing the reader's
|
||||
ch10 rigor work for them.
|
||||
|
||||
## Anti-patterns
|
||||
- **Using Spinning Up's PPO or TRPO as a paper baseline.** The page says explicitly to use
|
||||
Baselines for those. Under-powered baselines are the ch10 failure of "handicapping the
|
||||
baseline," even when it is unintentional.
|
||||
- **Comparing an on-policy `AverageEpRet` against an off-policy `AverageTestEpRet`** as if
|
||||
they were the same quantity. Use the plotter's `Performance` alias (ch6), which resolves
|
||||
per family.
|
||||
- **Reading a single-seed curve as a result.** These plots are 10 seeds precisely because
|
||||
fewer is not informative.
|
||||
|
||||
## Key Takeaways
|
||||
1. 3M timesteps, five MuJoCo environments, 10 seeds, both backends.
|
||||
2. DDPG, TD3, SAC: research-usable. VPG, TRPO, PPO: educational only — use Baselines.
|
||||
3. The weakness is attributed to two named omissions: observation normalization and
|
||||
normalized value regression targets.
|
||||
4. On-policy and off-policy performance are different measurements, not different numbers
|
||||
of the same measurement.
|
||||
5. Architecture defaults differ by family: (64,32)/tanh on-policy, (256,256)/relu off-policy.
|
||||
|
||||
## Connects To
|
||||
- **Ch 1**: "reasonably good" — this chapter is the quantification of that phrase.
|
||||
- **Ch 6**: the `Performance` alias, which exists because of the metric split described here.
|
||||
- **Ch 10**: the seed and fair-baseline standards this benchmark actually meets.
|
||||
- **Ch 14-19**: per-algorithm hyperparameter defaults referenced as "left at default settings."
|
||||
|
|
@ -0,0 +1,93 @@
|
|||
# Chapter 14: Vanilla Policy Gradient (VPG)
|
||||
|
||||
## Core Idea
|
||||
Push up the probabilities of actions that lead to higher return and push down the
|
||||
probabilities of actions that lead to lower return, until you arrive at the optimal policy.
|
||||
That is the whole idea underlying policy gradients, and VPG is its most direct expression.
|
||||
|
||||
## Quick Facts
|
||||
- **On-policy.**
|
||||
- Works with **discrete or continuous** action spaces.
|
||||
- The Spinning Up implementation **supports MPI parallelization**.
|
||||
|
||||
## Key Equations
|
||||
The advantage-weighted policy gradient (the end point of ch9):
|
||||
|
||||
```
|
||||
grad_theta J(pi_theta) = E_{tau ~ pi_theta}[ sum_{t=0}^{T} grad_theta log pi_theta(a_t|s_t) * A^{pi_theta}(s_t,a_t) ]
|
||||
```
|
||||
|
||||
Stochastic gradient ascent on policy performance:
|
||||
|
||||
```
|
||||
theta_{k+1} = theta_k + alpha * grad_theta J(pi_theta_k)
|
||||
```
|
||||
|
||||
**The notation mismatch worth knowing about:** `J(pi_theta)` here denotes the expected
|
||||
**finite-horizon undiscounted** return, but policy gradient implementations typically compute
|
||||
**advantage estimates based on the infinite-horizon discounted return** — despite otherwise
|
||||
using the finite-horizon undiscounted policy gradient formula. This is the ch7 blurring of
|
||||
the two return formulations, appearing in production code.
|
||||
|
||||
## Frameworks Introduced
|
||||
- **The VPG loop** (the template TRPO and PPO both modify, so learn it once):
|
||||
1. Collect a set of trajectories `D_k` by running `pi_k = pi(theta_k)` in the environment.
|
||||
2. Compute rewards-to-go `Rhat_t`.
|
||||
3. Compute advantage estimates `Ahat_t` (any method of advantage estimation) based on the
|
||||
current value function `V_{phi_k}`.
|
||||
4. Estimate the policy gradient:
|
||||
`ghat_k = (1/|D_k|) * sum_{tau in D_k} sum_t grad_theta log pi_theta(a_t|s_t)|_{theta_k} * Ahat_t`
|
||||
5. Update the policy by standard gradient ascent `theta_{k+1} = theta_k + alpha_k * ghat_k`,
|
||||
or another gradient ascent algorithm like Adam.
|
||||
6. Fit the value function by regression on mean-squared error:
|
||||
`phi_{k+1} = argmin_phi (1/(|D_k| T)) * sum sum (V_phi(s_t) - Rhat_t)^2`, typically via
|
||||
gradient descent.
|
||||
|
||||
## Exploration vs. Exploitation
|
||||
VPG trains a **stochastic policy in an on-policy way**, so it explores by sampling actions
|
||||
from the latest version of that policy. How random that is depends on both initial conditions
|
||||
and the training procedure. **Over training the policy typically becomes progressively less
|
||||
random**, because the update rule encourages exploiting rewards already found — **which may
|
||||
cause the policy to get trapped in local optima.** (This paragraph is identical for TRPO and
|
||||
PPO; it is a property of the on-policy stochastic-policy family, not of VPG specifically.)
|
||||
|
||||
## Code Examples
|
||||
Loading and using a trained PyTorch model:
|
||||
|
||||
```python
|
||||
ac = torch.load('path/to/model.pt')
|
||||
actions = ac.act(torch.as_tensor(obs, dtype=torch.float32))
|
||||
```
|
||||
|
||||
The TF1 saved graph exposes three keys: `x` (state input placeholder), `pi` (samples an
|
||||
action conditioned on `x`), and `v` (value estimate for states in `x`).
|
||||
|
||||
## Anti-patterns
|
||||
- **Expecting monotonic improvement.** VPG has none; the step size is unconstrained and a
|
||||
bad step can collapse performance. That collapse is what TRPO (ch15) exists to prevent.
|
||||
- **Under-fitting the value function.** Exercise 2.1 (ch12) shows a policy gradient agent
|
||||
with an untrained value function gets stuck early. The advantage estimate is only as good
|
||||
as `V_phi`.
|
||||
- **Reading the loss.** See ch9's worked example.
|
||||
|
||||
## Reference Tables
|
||||
|
||||
| Relevant paper | Why it is on the list |
|
||||
|----------------|----------------------|
|
||||
| Sutton et al. 2000, *Policy Gradient Methods for RL with Function Approximation* | A timeless classic of RL theory; contains references to the earlier work that led to modern policy gradients |
|
||||
| Schulman 2016(a), *Optimizing Expectations* | Chapter 2 is a lucid introduction to policy gradient theory, including pseudocode |
|
||||
| Duan et al. 2016, *Benchmarking Deep RL for Continuous Control* | A clear benchmark paper showing how VPG in the deep RL setting compares with other deep RL algorithms |
|
||||
| Schulman et al. 2016(b), *High Dimensional Continuous Control Using GAE* | Spinning Up's VPG uses GAE for computing the policy gradient |
|
||||
|
||||
## Key Takeaways
|
||||
1. VPG is the advantage-weighted policy gradient plus mean-squared value regression — nothing else.
|
||||
2. The six-step loop is the base template; TRPO changes step 5, PPO changes step 5 differently.
|
||||
3. Discrete or continuous, MPI-parallelizable, both backends available.
|
||||
4. Its weakness is step-size safety: nothing stops a single update from collapsing the policy.
|
||||
5. Exploration decays as a side effect of exploitation, with local optima the known risk.
|
||||
|
||||
## Connects To
|
||||
- **Ch 9**: the derivation that produces this exact gradient, and GAE.
|
||||
- **Ch 15**: TRPO, which constrains the step this chapter leaves unconstrained.
|
||||
- **Ch 12**: Exercise 2.1, on what happens when `V_phi` is not fit.
|
||||
- **Ch 13**: the benchmark caveat — VPG is educational-grade, not research-grade.
|
||||
|
|
@ -0,0 +1,122 @@
|
|||
# Chapter 15: Trust Region Policy Optimization (TRPO)
|
||||
|
||||
## Core Idea
|
||||
Take the **largest step possible to improve performance** while satisfying a constraint on
|
||||
how close the new and old policies are — measured in **KL-divergence between policies**, not
|
||||
distance in parameter space, because seemingly small parameter differences can produce very
|
||||
large performance differences.
|
||||
|
||||
## Quick Facts
|
||||
- **On-policy.**
|
||||
- Works with **discrete or continuous** action spaces.
|
||||
- **Supports MPI parallelization.**
|
||||
- **Tensorflow only** in Spinning Up; there is no PyTorch TRPO.
|
||||
|
||||
## Key Equations
|
||||
The theoretical update — maximize surrogate advantage subject to a KL trust region:
|
||||
|
||||
```
|
||||
theta_{k+1} = argmax_theta L(theta_k, theta)
|
||||
s.t. Dbar_KL(theta || theta_k) <= delta
|
||||
```
|
||||
|
||||
The **surrogate advantage** measures how `pi_theta` performs relative to the old policy
|
||||
*using data from the old policy* (an importance-weighted advantage):
|
||||
|
||||
```
|
||||
L(theta_k, theta) = E_{s,a ~ pi_theta_k}[ (pi_theta(a|s) / pi_theta_k(a|s)) * A^{pi_theta_k}(s,a) ]
|
||||
|
||||
Dbar_KL(theta || theta_k) = E_{s ~ pi_theta_k}[ D_KL( pi_theta(.|s) || pi_theta_k(.|s) ) ]
|
||||
```
|
||||
|
||||
Both the objective and the constraint are **zero at `theta = theta_k`**, and the gradient of
|
||||
the constraint is **also zero there**.
|
||||
|
||||
## Frameworks Introduced
|
||||
- **Taylor-expand to make it solvable.** Expand objective and constraint to leading order
|
||||
around `theta_k`:
|
||||
```
|
||||
L(theta_k, theta) ~= g^T (theta - theta_k)
|
||||
Dbar_KL(theta||theta_k) ~= 0.5 * (theta - theta_k)^T H (theta - theta_k)
|
||||
```
|
||||
**By happy coincidence, `g` — the gradient of the surrogate advantage at `theta_k` — is
|
||||
exactly equal to the policy gradient `grad_theta J(pi_theta)`.** Lagrangian duality then
|
||||
solves the approximate problem analytically:
|
||||
```
|
||||
theta_{k+1} = theta_k + sqrt( 2*delta / (g^T H^{-1} g) ) * H^{-1} g
|
||||
```
|
||||
**Stopping here would be exactly the Natural Policy Gradient.**
|
||||
|
||||
- **Backtracking line search — the fix that makes it TRPO.** Because the Taylor expansion
|
||||
introduces approximation error, that update may violate the KL constraint or fail to improve
|
||||
the surrogate advantage. So:
|
||||
```
|
||||
theta_{k+1} = theta_k + alpha^j * sqrt( 2*delta / (g^T H^{-1} g) ) * H^{-1} g
|
||||
```
|
||||
where `alpha` in (0,1) is the **backtracking coefficient** and `j` is the **smallest
|
||||
nonnegative integer** such that the new policy satisfies the KL constraint and produces a
|
||||
positive surrogate advantage.
|
||||
- When to use this pattern generally: whenever you solve an approximated problem and can
|
||||
cheaply check the exact condition, shrink the step until the exact condition holds.
|
||||
|
||||
- **Conjugate gradient instead of a matrix inverse.** Computing and storing `H^{-1}` is
|
||||
painfully expensive for networks with thousands or millions of parameters. TRPO solves
|
||||
`Hx = g` for `x = H^{-1} g` with conjugate gradient, which needs only a function computing
|
||||
the matrix-vector product `Hx`, never `H` itself:
|
||||
```
|
||||
Hx = grad_theta( (grad_theta Dbar_KL(theta || theta_k))^T x )
|
||||
```
|
||||
|
||||
## Reference Tables
|
||||
|
||||
| Hyperparameter | Role |
|
||||
|----------------|------|
|
||||
| `delta` | KL-divergence limit — the size of the trust region |
|
||||
| `alpha` | Backtracking coefficient, in (0,1) |
|
||||
| `K` | Maximum number of backtracking steps; `j` ranges over {0, 1, ..., K} |
|
||||
|
||||
The full loop is VPG's six steps with steps 5 replaced by: (a) conjugate gradient to compute
|
||||
`xhat_k ~= Hhat_k^{-1} ghat_k`, where `Hhat_k` is the Hessian of the sample average
|
||||
KL-divergence, then (b) the backtracking line search using
|
||||
`theta_{k+1} = theta_k + alpha^j * sqrt(2 delta / (xhat_k^T Hhat_k xhat_k)) * xhat_k`.
|
||||
|
||||
## Mental Models
|
||||
- **Trust region in policy space, not parameter space.** This is the whole insight. Ordinary
|
||||
policy gradient methods keep policies close in *parameter* space, but small parameter
|
||||
differences can mean very large performance differences — **so a single bad step can
|
||||
collapse the policy performance.** That is what makes large step sizes dangerous with
|
||||
vanilla policy gradients, and it is why VPG's sample efficiency suffers: it has to take
|
||||
small steps.
|
||||
- **TRPO buys back sample efficiency by making large steps safe**, and tends to improve
|
||||
performance **quickly and monotonically**.
|
||||
- Read the algorithm as **three nested approximations** — surrogate objective, Taylor
|
||||
expansion, conjugate gradient — each with a guard, the last being the line search.
|
||||
|
||||
## Exploration vs. Exploitation
|
||||
Identical to VPG: stochastic policy, on-policy sampling, randomness decaying over training as
|
||||
the update rule pushes toward exploiting known rewards, with local optima as the risk.
|
||||
|
||||
## Anti-patterns
|
||||
- **Skipping the line search** and shipping the analytic solution. That is Natural Policy
|
||||
Gradient, and the approximation error it inherits may violate the constraint or fail to
|
||||
improve anything.
|
||||
- **Forming `H` explicitly.** The matrix-vector product formulation exists because the matrix
|
||||
does not fit.
|
||||
- **Reaching for TRPO in PyTorch inside Spinning Up.** It does not exist; use PPO.
|
||||
|
||||
## Key Takeaways
|
||||
1. Constrain the *policy* change in KL, not the *parameter* change — the core contribution.
|
||||
2. Surrogate advantage is importance-weighted old-policy data; both it and the KL constraint
|
||||
vanish at `theta_k`.
|
||||
3. `g` (surrogate advantage gradient) equals the policy gradient — that is why VPG's machinery
|
||||
is reusable here.
|
||||
4. Analytic solution = Natural Policy Gradient; TRPO adds a backtracking line search on top.
|
||||
5. Conjugate gradient replaces the intractable `H^{-1}`.
|
||||
6. Three relevant papers: Schulman 2015 (original), Schulman 2016 (GAE, used here), Kakade and
|
||||
Langford 2002 (the theory motivating and connecting to TRPO's foundations).
|
||||
|
||||
## Connects To
|
||||
- **Ch 14**: the VPG loop TRPO modifies, and the collapse risk it removes.
|
||||
- **Ch 16**: PPO, which pursues the same goal with first-order methods.
|
||||
- **Ch 12**: Exercise 2.1 uses TRPO to demonstrate the value-fitting failure mode.
|
||||
- **Ch 10**: "monotonic improvement theory" — the optional math background this rests on.
|
||||
|
|
@ -0,0 +1,123 @@
|
|||
# Chapter 16: Proximal Policy Optimization (PPO)
|
||||
|
||||
## Core Idea
|
||||
PPO answers the same question as TRPO — how do you take the biggest possible improvement
|
||||
step without stepping so far that you cause performance collapse — but with **first-order
|
||||
methods and a few tricks** instead of a complex second-order method. It is significantly
|
||||
simpler to implement and empirically seems to perform at least as well.
|
||||
|
||||
## Quick Facts
|
||||
- **On-policy.**
|
||||
- Works with **discrete or continuous** action spaces.
|
||||
- **Supports MPI parallelization.**
|
||||
- Two variants exist; Spinning Up implements and documents **PPO-Clip**, the primary variant
|
||||
used at OpenAI.
|
||||
|
||||
## Frameworks Introduced
|
||||
- **The two PPO variants:**
|
||||
- **PPO-Penalty** approximately solves a KL-constrained update like TRPO, but **penalizes
|
||||
the KL-divergence in the objective** instead of making it a hard constraint, and
|
||||
**automatically adjusts the penalty coefficient** over training so it stays appropriately
|
||||
scaled.
|
||||
- **PPO-Clip** has **no KL term in the objective and no constraint at all.** It relies on
|
||||
specialized **clipping in the objective function** to remove the incentive for the new
|
||||
policy to get far from the old.
|
||||
|
||||
- **The PPO-Clip objective.** Update by
|
||||
`theta_{k+1} = argmax_theta E_{s,a ~ pi_theta_k}[ L(s,a,theta_k,theta) ]`, typically with
|
||||
**multiple steps of (usually minibatch) SGD**. The published form:
|
||||
```
|
||||
L = min( ratio * A, clip(ratio, 1-eps, 1+eps) * A )
|
||||
where ratio = pi_theta(a|s) / pi_theta_k(a|s)
|
||||
```
|
||||
and `eps` is a small hyperparameter roughly saying how far the new policy may go.
|
||||
|
||||
- **The simplified equivalent form** (this is the version Spinning Up actually implements,
|
||||
and it is much easier to reason about):
|
||||
```
|
||||
L = min( ratio * A, g(eps, A) )
|
||||
where g(eps, A) = (1 + eps) * A if A >= 0
|
||||
(1 - eps) * A if A < 0
|
||||
```
|
||||
|
||||
## Worked Example
|
||||
**Why clipping removes the incentive to move far — the two cases.**
|
||||
|
||||
**Advantage positive.** The term reduces to `min(ratio, 1+eps) * A`. Because `A > 0`, the
|
||||
objective increases as the action becomes more likely, i.e. as `pi_theta(a|s)` increases.
|
||||
But the `min` caps how much. Once `pi_theta(a|s) > (1+eps) * pi_theta_k(a|s)`, the min kicks
|
||||
in and the term hits a ceiling of `(1+eps) * A`. **The new policy does not benefit by going
|
||||
far away from the old policy.**
|
||||
|
||||
**Advantage negative.** The term reduces to `max(ratio, 1-eps) * A`. Because `A < 0`, the
|
||||
objective increases as the action becomes *less* likely, i.e. as `pi_theta(a|s)` decreases.
|
||||
The `max` caps how much: once `pi_theta(a|s) < (1-eps) * pi_theta_k(a|s)`, it hits a ceiling
|
||||
of `(1-eps) * A`. **Again: the new policy does not benefit by going far away from the old.**
|
||||
|
||||
So **clipping serves as a regularizer by removing incentives for the policy to change
|
||||
dramatically**, and `eps` corresponds to how far the new policy can go while still profiting
|
||||
from the objective.
|
||||
|
||||
**The honest caveat, and Spinning Up's answer.** Clipping goes a long way toward reasonable
|
||||
updates, but **it is still possible to end up with a new policy too far from the old**, and
|
||||
different PPO implementations use a variety of tricks to stave this off. Spinning Up uses a
|
||||
particularly simple one: **early stopping — if the mean KL-divergence of the new policy from
|
||||
the old grows beyond a threshold, stop taking gradient steps.** Note that this reintroduces a
|
||||
KL measurement into an algorithm defined by not having one; the clip is the incentive
|
||||
mechanism, the KL check is the safety net.
|
||||
|
||||
## Reference Tables
|
||||
|
||||
The PPO-Clip loop, as a diff against VPG's six steps (ch14):
|
||||
|
||||
| Step | VPG | PPO-Clip |
|
||||
|------|-----|----------|
|
||||
| 1-3 | Collect trajectories, rewards-to-go, advantage estimates | identical |
|
||||
| 4-5 | Estimate `ghat_k`, single ascent step | **Maximize the PPO-Clip objective, typically via multiple steps of stochastic gradient ascent with Adam** |
|
||||
| 6 | Fit `V_phi` by MSE regression on `Rhat_t` | identical |
|
||||
|
||||
The key structural difference from VPG and TRPO: **multiple gradient steps per batch of data**,
|
||||
which is where the sample-efficiency gain comes from and why the policy needs restraining at all.
|
||||
|
||||
| Relevant paper | Why it is on the list |
|
||||
|----------------|----------------------|
|
||||
| Schulman et al. 2017, *Proximal Policy Optimization Algorithms* | The original PPO paper |
|
||||
| Schulman et al. 2016, *High Dimensional Continuous Control Using GAE* | Spinning Up's PPO uses GAE for the policy gradient |
|
||||
| Heess et al. 2017, *Emergence of Locomotion Behaviours in Rich Environments* | Large-scale empirical analysis of behaviors learned by PPO agents in complex environments — though it uses PPO-Penalty, not PPO-Clip |
|
||||
|
||||
## Exploration vs. Exploitation
|
||||
Identical to VPG and TRPO: stochastic policy, on-policy sampling, decaying randomness, local
|
||||
optima as the failure mode.
|
||||
|
||||
## Mental Models
|
||||
- **PPO trades a hard guarantee for implementability.** TRPO enforces the trust region;
|
||||
PPO removes the *incentive* to leave it. That is a weaker statement, and PPO's practical
|
||||
dominance says the weaker statement is usually enough.
|
||||
- **The clip is one-sided per case.** It only ever caps the *beneficial* direction. It never
|
||||
penalizes moving the wrong way — it just stops paying you for moving further the right way.
|
||||
- Careful with terminology: **ch9 notes PPO is often called a policy gradient algorithm,
|
||||
though this is slightly inaccurate** — it maximizes a surrogate objective rather than
|
||||
following `grad J` directly.
|
||||
|
||||
## Anti-patterns
|
||||
- **Implementing PPO-Clip and calling the result PPO without saying which variant.** Two
|
||||
public implementations named "PPO" (ModularRL, rllab) implement PPO-**Penalty**; comparing
|
||||
against them without noticing is a real reproduction hazard the docs explicitly flag.
|
||||
- **Relying on the clip alone at large step counts.** More SGD steps per batch means more
|
||||
opportunity to drift; that is precisely why the early-stopping KL check exists.
|
||||
- **Tuning `eps` as if it were a learning rate.** It bounds policy change per update; the
|
||||
number of SGD steps and the learning rate interact with it.
|
||||
|
||||
## Key Takeaways
|
||||
1. Same goal as TRPO, first-order machinery: simpler to implement, at least as good empirically.
|
||||
2. PPO-Clip (no KL term, no constraint) is the OpenAI default and Spinning Up's implementation.
|
||||
3. The simplified `g(eps, A)` form makes the mechanism obvious: a ceiling in the profitable
|
||||
direction, in both the positive- and negative-advantage cases.
|
||||
4. Clipping is an incentive change, not a guarantee — Spinning Up backs it with KL early stopping.
|
||||
5. Multiple minibatch SGD steps per batch is the structural difference from VPG and TRPO.
|
||||
|
||||
## Connects To
|
||||
- **Ch 15**: TRPO's hard constraint, the thing PPO replaces.
|
||||
- **Ch 9**: the surrogate-objective framing and why "policy gradient algorithm" is imprecise here.
|
||||
- **Ch 12**: Exercise 1.2 asks you to implement the MLP diagonal Gaussian policy for PPO.
|
||||
- **Ch 13**: use OpenAI Baselines' PPO, not this one, for research comparisons.
|
||||
|
|
@ -0,0 +1,122 @@
|
|||
# Chapter 17: Deep Deterministic Policy Gradient (DDPG)
|
||||
|
||||
## Core Idea
|
||||
DDPG concurrently learns a Q-function and a policy: it uses off-policy data and the Bellman
|
||||
equation to learn the Q-function, and uses the Q-function to learn the policy. It exists
|
||||
because `max_a Q*(s,a)` — trivial over discrete actions — is intractable over continuous ones,
|
||||
and DDPG replaces that max with a learned, differentiable approximation.
|
||||
|
||||
## Quick Facts
|
||||
- **Off-policy.**
|
||||
- **Continuous action spaces only.**
|
||||
- Can be thought of as **deep Q-learning for continuous action spaces.**
|
||||
- The Spinning Up implementation **does not support parallelization.**
|
||||
|
||||
## Frameworks Introduced
|
||||
- **The continuous-argmax problem, and the substitution that solves it.** With finitely many
|
||||
discrete actions, the max poses no problem — compute Q for each and compare, which also
|
||||
immediately hands you the maximizing action. With a continuous action space you cannot
|
||||
exhaustively evaluate, and a general optimization subroutine would have to run **every time
|
||||
the agent wants to take an action**, which is unacceptable. But because the action space is
|
||||
continuous, `Q*(s,a)` is presumed **differentiable with respect to the action** — so you can
|
||||
learn a policy `mu(s)` by gradient ascent and approximate
|
||||
`max_a Q(s,a) ~= Q(s, mu(s))`.
|
||||
- When to use this pattern: any time an inner argmax over a continuous variable sits inside
|
||||
an outer loop. Amortize it into a learned function.
|
||||
|
||||
- **Mean-Squared Bellman Error (MSBE)** — the loss that all deep Q-learning minimizes:
|
||||
```
|
||||
L(phi, D) = E_{(s,a,r,s',d) ~ D}[ ( Q_phi(s,a) - ( r + gamma*(1-d)*max_a' Q_phi(s',a') ) )^2 ]
|
||||
```
|
||||
`d` indicates whether `s'` is terminal, using the Python convention of `True` as 1: when
|
||||
`d == True` the Q-function should show the agent gets no additional reward after this state.
|
||||
|
||||
- **Trick One: Replay Buffers.** All standard algorithms training a deep network to approximate
|
||||
`Q*(s,a)` use an experience replay buffer — the set `D` of previous experiences.
|
||||
- **The buffer size is a real tuning decision, in both directions:** it should be large
|
||||
enough to contain a wide range of experiences, but keeping everything may not be good
|
||||
either. **Use only the very-most recent data and you overfit to it and things break; use
|
||||
too much experience and you may slow down learning.**
|
||||
- **Why old data is admissible at all** — this is the cleanest statement of what off-policy
|
||||
means: **the Bellman equation does not care which transition tuples are used, how the
|
||||
actions were selected, or what happens after a given transition**, because the optimal
|
||||
Q-function must satisfy it for *all* possible transitions. So any transition you have ever
|
||||
experienced is fair game for MSBE minimization.
|
||||
|
||||
- **Trick Two: Target Networks.** The target `r + gamma*(1-d)*max_a' Q_phi(s',a')` depends on
|
||||
the same parameters `phi` being trained, which makes MSBE minimization unstable. The fix is
|
||||
a second network that lags the first, with parameters `phi_targ`.
|
||||
- **DQN-style**: copy the main network into the target every fixed number of steps.
|
||||
- **DDPG-style**: **polyak averaging**, once per main network update:
|
||||
`phi_targ <- rho * phi_targ + (1-rho) * phi`, with `rho` in (0,1), usually close to 1.
|
||||
(`rho` is called `polyak` in the code.)
|
||||
|
||||
- **DDPG detail: the max in the target.** Since the max over continuous actions is the original
|
||||
problem, DDPG uses a **target policy network** `mu_theta_targ` to compute an action that
|
||||
approximately maximizes `Q_phi_targ`, found the same way as the target Q-function — by
|
||||
polyak averaging the policy parameters over training. Final Q-loss:
|
||||
```
|
||||
L(phi,D) = E[ ( Q_phi(s,a) - ( r + gamma*(1-d)*Q_phi_targ(s', mu_theta_targ(s')) ) )^2 ]
|
||||
```
|
||||
|
||||
- **Policy learning is one line**: maximize `E_{s~D}[ Q_phi(s, mu_theta(s)) ]` by gradient
|
||||
ascent **with respect to policy parameters only** — the Q-function parameters are treated
|
||||
as constants.
|
||||
|
||||
## Exploration vs. Exploitation
|
||||
DDPG trains a **deterministic policy off-policy**. Because the policy is deterministic,
|
||||
on-policy exploration would probably not try a wide enough variety of actions early on to
|
||||
find useful learning signal. So **noise is added to actions at training time**.
|
||||
- The original DDPG paper recommended **time-correlated OU noise**, but **more recent results
|
||||
suggest uncorrelated, mean-zero Gaussian noise works perfectly well** — and since it is
|
||||
simpler, it is preferred.
|
||||
- You may reduce the noise scale over training to get higher-quality data. **Spinning Up does
|
||||
not do this and keeps the noise scale fixed throughout.**
|
||||
- **At test time, no noise is added** — that is how you see how well the policy exploits what
|
||||
it has learned.
|
||||
- **`start_steps` trick**: for a fixed number of steps at the beginning, the agent takes
|
||||
actions sampled from a **uniform random distribution over valid actions**, then reverts to
|
||||
normal DDPG exploration.
|
||||
|
||||
## Reference Tables
|
||||
|
||||
The DDPG loop:
|
||||
|
||||
| Step | Action |
|
||||
|------|--------|
|
||||
| Init | Set target parameters equal to main: `theta_targ <- theta`, `phi_targ <- phi` |
|
||||
| Act | Observe `s`, select `a = clip(mu_theta(s) + epsilon, a_Low, a_High)`, `epsilon ~ N` |
|
||||
| Store | Execute `a`, observe `s'`, `r`, `d`; store `(s,a,r,s',d)` in `D`; reset if terminal |
|
||||
| Update | Sample a batch `B` from `D`; compute targets `y = r + gamma*(1-d)*Q_phi_targ(s', mu_theta_targ(s'))` |
|
||||
| | One gradient descent step on `(Q_phi(s,a) - y)^2` averaged over `B` |
|
||||
| | One gradient ascent step on `Q_phi(s, mu_theta(s))` averaged over `B` |
|
||||
| | Polyak-update both target networks |
|
||||
|
||||
| Relevant paper | Why it is on the list |
|
||||
|----------------|----------------------|
|
||||
| Silver et al. 2014, *Deterministic Policy Gradient Algorithms* | Establishes the theory underlying deterministic policy gradients (DPG) |
|
||||
| Lillicrap et al. 2016, *Continuous Control With Deep RL* | Adapts the theoretically-grounded DPG algorithm to the deep RL setting, giving DDPG |
|
||||
|
||||
## Anti-patterns
|
||||
- **DDPG on a discrete action space.** The entire design is the continuous-action workaround.
|
||||
- **Trusting the original paper's full recipe.** Ch10 uses DDPG as its example: the paper
|
||||
suggests a complex architecture, an initialization scheme and batch normalization, none of
|
||||
which are strictly necessary, and some of the best DDPG results use simpler networks.
|
||||
- **Assuming DDPG is stable.** It is **frequently brittle with respect to hyperparameters and
|
||||
other kinds of tuning** — that is the premise of ch18.
|
||||
- **Shape bugs in the Q-function.** Exercise 2.2 (ch12) plants exactly one in DDPG's critic
|
||||
and it silently ruins learning through broadcasting.
|
||||
|
||||
## Key Takeaways
|
||||
1. `Q(s, mu(s))` replaces `max_a Q(s,a)` — the whole reason DDPG exists.
|
||||
2. Replay buffers are licensed by the Bellman equation's indifference to how data was collected.
|
||||
3. Target networks fix the moving-target instability; DDPG uses polyak averaging, DQN uses copies.
|
||||
4. Two target networks in DDPG: target Q *and* target policy.
|
||||
5. Explore with additive Gaussian action noise plus uniform-random `start_steps`; no noise at test.
|
||||
6. Replay buffer size is a genuine trade-off, wrong in both directions.
|
||||
|
||||
## Connects To
|
||||
- **Ch 7**: `a*(s) = argmax_a Q*(s,a)` and the Bellman optimality equation this is built on.
|
||||
- **Ch 8**: DDPG as the named interpolation between policy optimization and Q-learning.
|
||||
- **Ch 18**: TD3, the three tricks that fix DDPG's brittleness.
|
||||
- **Ch 12**: Exercise 2.2, the silent bug planted in this algorithm.
|
||||
|
|
@ -0,0 +1,107 @@
|
|||
# Chapter 18: Twin Delayed DDPG (TD3)
|
||||
|
||||
## Core Idea
|
||||
DDPG's common failure mode is that **the learned Q-function begins to dramatically
|
||||
overestimate Q-values, which then breaks the policy, because the policy exploits the errors
|
||||
in the Q-function.** TD3 fixes this with three tricks, and nothing else changes.
|
||||
|
||||
## Quick Facts
|
||||
- **Off-policy.**
|
||||
- **Continuous action spaces only.**
|
||||
- The Spinning Up implementation **does not support parallelization.**
|
||||
|
||||
## Frameworks Introduced
|
||||
- **Trick One: Clipped Double-Q Learning.** Learn **two** Q-functions instead of one (hence
|
||||
"twin") and **use the smaller of the two Q-values to form the targets** in the Bellman error
|
||||
loss functions. Both are then regressed toward that single shared target:
|
||||
```
|
||||
y(r,s',d) = r + gamma*(1-d) * min_{i=1,2} Q_{phi_i,targ}(s', a'(s'))
|
||||
L(phi_1, D) = E[ (Q_phi_1(s,a) - y)^2 ]
|
||||
L(phi_2, D) = E[ (Q_phi_2(s,a) - y)^2 ]
|
||||
```
|
||||
**Using the smaller Q-value for the target, and regressing toward that, helps fend off
|
||||
overestimation** — a pessimistic estimate cannot be exploited the way an optimistic one can.
|
||||
|
||||
- **Trick Two: "Delayed" Policy Updates.** Update the policy (and the target networks) **less
|
||||
frequently than the Q-function**. **The paper recommends one policy update for every two
|
||||
Q-function updates.** This **damps the volatility that normally arises in DDPG because of
|
||||
how a policy update changes the target.** Controlled by `policy_delay`.
|
||||
|
||||
- **Trick Three: Target Policy Smoothing.** Add **clipped** noise to each dimension of the
|
||||
target action, then clip the result back into the valid action range:
|
||||
```
|
||||
a'(s') = clip( mu_theta_targ(s') + clip(epsilon, -c, c), a_Low, a_High ), epsilon ~ N(0, sigma)
|
||||
```
|
||||
**This is a regularizer.** It addresses a specific DDPG failure: **if the Q-function
|
||||
approximator develops an incorrect sharp peak for some actions, the policy will quickly
|
||||
exploit that peak and then have brittle or incorrect behavior.** Smoothing Q over similar
|
||||
actions averts it.
|
||||
|
||||
- **Policy learning is unchanged from DDPG except in which critic it uses:** maximize
|
||||
`E_{s~D}[ Q_phi_1(s, mu_theta(s)) ]` — **just the first Q-function.** (SAC differs here; see
|
||||
ch19.)
|
||||
|
||||
## Reference Tables
|
||||
|
||||
| Trick | Fixes | Mechanism |
|
||||
|-------|-------|-----------|
|
||||
| Clipped double-Q | Q-value overestimation | Two critics, `min` of the two forms the shared target |
|
||||
| Delayed policy updates | Volatility from policy updates moving the target | One policy + target update per `policy_delay` critic updates |
|
||||
| Target policy smoothing | Policy exploiting sharp incorrect peaks in Q | Clipped Gaussian noise on the target action, then clip to action bounds |
|
||||
|
||||
| Hyperparameter | Role |
|
||||
|----------------|------|
|
||||
| `sigma` | Std dev of the target-smoothing noise |
|
||||
| `c` | Clip bound on that noise |
|
||||
| `policy_delay` | Q-updates per policy update; paper recommends 2 |
|
||||
| `rho` (`polyak`) | Target network averaging coefficient, inherited from DDPG |
|
||||
| `start_steps` | Initial uniform-random action steps, inherited from DDPG |
|
||||
|
||||
The loop is DDPG's, with two changes inside the update block: compute the smoothed target
|
||||
action `a'(s')` first, update **both** Q-functions toward the clipped-double-Q target, and
|
||||
then — **only when `j mod policy_delay == 0`** — take the policy ascent step and polyak-update
|
||||
all three target networks.
|
||||
|
||||
## Exploration vs. Exploitation
|
||||
Identical to DDPG: deterministic policy explored with **uncorrelated mean-zero Gaussian action
|
||||
noise** at training time, optionally decayed (Spinning Up keeps it fixed), **no noise at test
|
||||
time**, and a `start_steps` phase of uniform random actions at the beginning.
|
||||
|
||||
Note the distinction that is easy to blur: **exploration noise is added to the acting policy;
|
||||
target policy smoothing noise is added to the target action inside the update.** Two different
|
||||
noises, two different jobs.
|
||||
|
||||
## Mental Models
|
||||
- **All three tricks are forms of pessimism or patience.** Take the smaller estimate, wait
|
||||
longer before acting on the critic, and refuse to believe a Q-value that does not hold up
|
||||
under a small action perturbation. Overestimation is the disease; conservatism is the cure.
|
||||
- **Think of the policy as an adversary against your critic's errors.** Anything the critic
|
||||
gets wrong in the optimistic direction is exactly what the policy will find. This is the
|
||||
same failure shape as model bias in ch8's model-based methods.
|
||||
- **"Together, these three tricks result in substantially improved performance over baseline
|
||||
DDPG"** — they are presented as a package, and the exercise (ch12, 1.3) has you implement
|
||||
the losses that contain all three.
|
||||
|
||||
## Anti-patterns
|
||||
- **Implementing only clipped double-Q and calling it TD3.** The three tricks address three
|
||||
distinct failure modes and the paper's result is for the package.
|
||||
- **Using `min` over the two critics for the *policy* loss.** TD3 uses `Q_phi_1` alone for the
|
||||
policy; using the min there is a SAC choice, not a TD3 one.
|
||||
- **Updating the target networks on every critic step.** The target updates are inside the
|
||||
`policy_delay` branch, along with the policy update.
|
||||
|
||||
## Key Takeaways
|
||||
1. TD3 = DDPG + clipped double-Q + delayed policy updates + target policy smoothing.
|
||||
2. The disease being treated is Q-value overestimation, which the policy actively exploits.
|
||||
3. `min` of two critics forms the target; the policy maximizes `Q_phi_1` only.
|
||||
4. Policy and target-network updates both live behind `policy_delay` (recommended 2).
|
||||
5. Target smoothing noise is clipped twice: clip the noise to `[-c, c]`, then clip the
|
||||
resulting action to the valid range.
|
||||
6. Relevant paper: Fujimoto et al, 2018, *Addressing Function Approximation Error in
|
||||
Actor-Critic Methods.*
|
||||
|
||||
## Connects To
|
||||
- **Ch 17**: DDPG, whose brittleness is TD3's entire premise.
|
||||
- **Ch 19**: SAC, which borrows clipped double-Q and gets smoothing for free from stochasticity.
|
||||
- **Ch 12**: Exercise 1.3 asks you to write exactly these losses.
|
||||
- **Ch 13**: TD3 is one of the three research-grade Spinning Up implementations.
|
||||
|
|
@ -0,0 +1,133 @@
|
|||
# Chapter 19: Soft Actor-Critic (SAC)
|
||||
|
||||
## Core Idea
|
||||
SAC optimizes a **stochastic** policy in an **off-policy** way, forming a bridge between
|
||||
stochastic policy optimization and DDPG-style approaches. Its central feature is **entropy
|
||||
regularization**: the policy is trained to maximize a trade-off between expected return and
|
||||
the randomness of the policy.
|
||||
|
||||
## Quick Facts
|
||||
- **Off-policy.**
|
||||
- The version implemented here is **continuous action spaces only** — an alternate version,
|
||||
with a slightly changed policy update rule, can handle discrete action spaces.
|
||||
- **Does not support parallelization.**
|
||||
- SAC is **not a direct successor to TD3** — they were published roughly concurrently — but it
|
||||
incorporates the clipped double-Q trick, and its inherent policy stochasticity gives it
|
||||
something like target policy smoothing for free.
|
||||
|
||||
## Frameworks Introduced
|
||||
- **Entropy-regularized RL.** Entropy `H(P) = E_{x~P}[-log P(x)]` says how random a random
|
||||
variable is: a coin that almost always comes up heads has low entropy, a fair coin has high
|
||||
entropy. The agent gets a bonus reward at each timestep proportional to policy entropy,
|
||||
changing the RL problem to:
|
||||
```
|
||||
pi* = argmax_pi E_{tau~pi}[ sum_t gamma^t ( R(s_t,a_t,s_{t+1}) + alpha * H(pi(.|s_t)) ) ]
|
||||
```
|
||||
`alpha > 0` is the trade-off coefficient. **This has a close connection to the
|
||||
exploration-exploitation trade-off: increasing entropy results in more exploration, which
|
||||
can accelerate learning later on, and can prevent the policy from prematurely converging to
|
||||
a bad local optimum.**
|
||||
|
||||
- **The modified value functions.** `V^pi` includes the entropy bonus from **every** timestep;
|
||||
`Q^pi` includes it from every timestep **except the first**. They connect by
|
||||
```
|
||||
V^pi(s) = E_{a~pi}[ Q^pi(s,a) ] + alpha * H(pi(.|s))
|
||||
= E_{a~pi}[ Q^pi(s,a) - alpha * log pi(a|s) ]
|
||||
```
|
||||
**This setup is a little bit arbitrary** — you could instead have `Q^pi` include the first
|
||||
timestep's bonus — and **the choice of definition varies slightly across papers on the
|
||||
subject.** Check before comparing equations across sources.
|
||||
|
||||
- **What SAC keeps from TD3, and what it changes.** SAC learns a policy `pi_theta` and two
|
||||
Q-functions `Q_phi_1`, `Q_phi_2`.
|
||||
- **Same as TD3**: both Q-functions learned by MSBE minimization regressing to a **single
|
||||
shared target**; the target computed using **target Q-networks obtained by polyak
|
||||
averaging**; the **clipped double-Q trick**.
|
||||
- **Different from TD3**: (1) the target includes an **entropy regularization term**;
|
||||
(2) the **next-state actions in the target come from the current policy, not a target
|
||||
policy**; (3) **no explicit target policy smoothing** — TD3 trains a deterministic policy
|
||||
and so needs added noise to smooth, whereas **SAC's policy stochasticity is sufficient to
|
||||
get a similar effect**.
|
||||
|
||||
- **The Q-loss.** Rewrite the entropy-regularized Bellman equation using
|
||||
`H = -log pi`, approximate the expectation with samples, and take the min over the two
|
||||
critics:
|
||||
```
|
||||
y(r,s',d) = r + gamma*(1-d) * ( min_{j=1,2} Q_{phi_j,targ}(s', a'~) - alpha * log pi_theta(a'~|s') )
|
||||
where a'~ ~ pi_theta(.|s')
|
||||
```
|
||||
**Notation matters here:** `r` and `s'` come from the **replay buffer**, but `a'~` must be
|
||||
**sampled fresh from the current policy** — hence the tilde. Mixing these up is a silent bug.
|
||||
|
||||
- **Policy learning via the reparameterization trick.** The policy should maximize `V^pi(s)`,
|
||||
i.e. `E_{a~pi}[ Q^pi(s,a) - alpha log pi(a|s) ]`. The expectation's distribution depends on
|
||||
the policy parameters, which is the pain point; the reparameterization trick rewrites it as
|
||||
an expectation over **noise**, which has no parameter dependence. Using a **squashed Gaussian
|
||||
policy**:
|
||||
```
|
||||
a~_theta(s, xi) = tanh( mu_theta(s) + sigma_theta(s) * xi ), xi ~ N(0, I)
|
||||
```
|
||||
giving the policy objective
|
||||
```
|
||||
max_theta E_{s~D, xi~N}[ min_{j=1,2} Q_phi_j(s, a~_theta(s,xi)) - alpha * log pi_theta(a~_theta(s,xi)|s) ]
|
||||
```
|
||||
**almost the same as DDPG and TD3 policy optimization, except for the min-double-Q trick,
|
||||
the stochasticity, and the entropy term.** Note that unlike TD3 (which uses `Q_phi_1` only),
|
||||
**SAC uses the min of the two approximators in the policy loss too.**
|
||||
|
||||
## Reference Tables
|
||||
|
||||
**Two ways SAC's policy differs from the VPG/TRPO/PPO policies** — both are load-bearing:
|
||||
|
||||
| Difference | SAC | VPG / TRPO / PPO |
|
||||
|-----------|-----|------------------|
|
||||
| **Squashing function** | `tanh` ensures actions are bounded to a finite range. Before the tanh the policy is a factored Gaussian; after it, it is not. Log-probabilities are still computable in closed form (see the paper appendix) | No squashing |
|
||||
| **Std dev parameterization** | log stds are **outputs of the neural network**, so they depend on state in a complex way | log stds are **state-independent parameter vectors** |
|
||||
|
||||
**SAC with state-independent log std devs, in the authors' experience, did not work.**
|
||||
|
||||
| Variant | Status |
|
||||
|---------|--------|
|
||||
| Fixed entropy coefficient `alpha` | What Spinning Up implements, for simplicity |
|
||||
| Entropy-constrained (varies `alpha` over training) | **Generally preferred by practitioners** |
|
||||
| Older SAC that also learns a value function `V_psi` | Superseded; the docs cover the modern version that omits it |
|
||||
|
||||
## Exploration vs. Exploitation
|
||||
SAC trains a stochastic policy with entropy regularization and explores **on-policy**.
|
||||
**`alpha` explicitly controls the explore-exploit trade-off** — higher means more exploration,
|
||||
lower means more exploitation. **The right coefficient may vary from environment to environment
|
||||
and could require careful tuning.** At test time, **remove the stochasticity and use the mean
|
||||
action instead of a sample**; this tends to improve performance over the stochastic policy.
|
||||
(This is why `test_policy` has a SAC-only `--deterministic` flag — see ch5.) `start_steps`
|
||||
uniform-random exploration at the beginning applies here too.
|
||||
|
||||
## Anti-patterns
|
||||
- **Evaluating SAC with the stochastic policy.** The correct evaluation policy is the
|
||||
deterministic mean; `test_policy` defaults to the stochastic one and needs `-d`.
|
||||
- **Using state-independent log stds in SAC** because that is what PPO does. It did not work.
|
||||
- **Sampling the target's next action from the replay buffer.** It must come fresh from the
|
||||
current policy.
|
||||
- **Assuming SAC's `Q^pi` definition matches another paper's.** The placement of the
|
||||
first-timestep entropy bonus is a convention that varies.
|
||||
- **Treating `alpha` as a set-and-forget constant across environments.** It is the
|
||||
explore-exploit dial and is environment-specific; the entropy-constrained variant exists
|
||||
because tuning it by hand is hard.
|
||||
|
||||
## Key Takeaways
|
||||
1. Entropy regularization is the defining feature: maximize return plus `alpha` times policy
|
||||
entropy, which directly controls exploration.
|
||||
2. SAC keeps TD3's clipped double-Q and polyak targets, adds an entropy term to the target,
|
||||
uses the current policy (not a target policy) for next-state actions, and needs no explicit
|
||||
smoothing.
|
||||
3. The squashed Gaussian plus reparameterization trick makes the stochastic policy
|
||||
differentiable end to end.
|
||||
4. SAC uses `min` over both critics in the policy loss; TD3 uses only `Q_phi_1`.
|
||||
5. State-dependent log stds are required — state-independent ones did not work.
|
||||
6. Evaluate with the mean action. Spinning Up ships the fixed-`alpha` variant; practitioners
|
||||
generally prefer the entropy-constrained one.
|
||||
|
||||
## Connects To
|
||||
- **Ch 18**: TD3, whose clipped double-Q trick SAC borrows and whose smoothing it obviates.
|
||||
- **Ch 7**: the diagonal Gaussian policy and the two ways of parameterizing log stds.
|
||||
- **Ch 5**: the SAC-only `--deterministic` flag on `test_policy`.
|
||||
- **Ch 10**: the reparameterization trick is on the required deep-learning background list.
|
||||
|
|
@ -0,0 +1,99 @@
|
|||
# Chapter 20: Logger, MPI Tools and Run Utils
|
||||
|
||||
## Core Idea
|
||||
The three utility modules are the only code shared between algorithms — logging (which is
|
||||
also model saving and config recording), MPI data-parallelism, and ExperimentGrid — and
|
||||
each has a small fixed usage pattern worth memorizing.
|
||||
|
||||
## Frameworks Introduced
|
||||
- **`Logger` and `EpochLogger`.** `Logger` carries the basic functionality: saving
|
||||
diagnostics, hyperparameter configurations, the state of a training run, and the trained
|
||||
model. `EpochLogger` adds a thin layer that makes it easy to track the **average, standard
|
||||
deviation, min and max** of a diagnostic over each epoch **and across MPI workers**.
|
||||
**All Spinning Up algorithm implementations use an EpochLogger.**
|
||||
- The three-call pattern: **`store`** accumulates values into internal state,
|
||||
**`log_tabular`** computes the statistics over everything stored, **`dump_tabular`**
|
||||
writes to file and stdout. **The internal state is wiped clean after `log_tabular`,
|
||||
to prevent leakage into the next epoch's statistics.**
|
||||
|
||||
- **The MPI + PyTorch pattern** — three steps, in this order:
|
||||
1. At the beginning of the training script, call **`setup_pytorch_for_mpi()`**. This is the
|
||||
fix for a real problem: each separate process's PyTorch instance tries to grab too many
|
||||
threads and they clobber each other.
|
||||
2. After constructing a PyTorch module, call **`sync_params(module)`**.
|
||||
3. During gradient descent, call **`mpi_avg_grads`** after the backward pass and before the
|
||||
optimizer step.
|
||||
- The two main ingredients are therefore **syncing parameters** and **averaging gradients
|
||||
before they are used by the adaptive optimizer** — the order matters, because averaging
|
||||
after the optimizer step would give each worker a different adaptive state.
|
||||
|
||||
- **ExperimentGrid** — a tool for hyperparameter ablations, based on but simpler than rllab's
|
||||
VariantGenerator. `eg.add(param_name, values, shorthand, in_name)` then
|
||||
`eg.run(thunk, **run_kwargs)`; `ExperimentGrid.run` uses `call_experiment` to launch the
|
||||
thunk. See ch4 for the full usage.
|
||||
|
||||
## Code Examples
|
||||
The EpochLogger statistic pattern:
|
||||
|
||||
```python
|
||||
from spinup.utils.logx import EpochLogger
|
||||
epoch_logger = EpochLogger()
|
||||
for i in range(10):
|
||||
epoch_logger.store(Test=i)
|
||||
epoch_logger.log_tabular('Test', with_min_and_max=True)
|
||||
epoch_logger.dump_tabular()
|
||||
```
|
||||
|
||||
```
|
||||
-------------------------------------
|
||||
| AverageTest | 4.5 |
|
||||
| StdTest | 2.87 |
|
||||
| MaxTest | 9 |
|
||||
| MinTest | 0 |
|
||||
-------------------------------------
|
||||
```
|
||||
|
||||
The MPI gradient-averaging step, in place:
|
||||
|
||||
```python
|
||||
optimizer.zero_grad()
|
||||
loss = compute_loss(module)
|
||||
loss.backward()
|
||||
mpi_avg_grads(module) # averages gradient buffers across MPI processes
|
||||
optimizer.step()
|
||||
```
|
||||
|
||||
## Reference Tables
|
||||
|
||||
| Module | Contents |
|
||||
|--------|----------|
|
||||
| `spinup.utils.logx` | `Logger`, `EpochLogger` |
|
||||
| `spinup.utils.mpi_tools` | Core MPI utilities |
|
||||
| `spinup.utils.mpi_pytorch` | `setup_pytorch_for_mpi`, `sync_params`, `mpi_avg_grads` |
|
||||
| `spinup.utils.mpi_tf` | AdamOptimizer across MPI processes. **Explicitly "a bit hacky"** — for something more sophisticated and general-purpose, the docs point to horovod |
|
||||
| `spinup.utils.run_utils` | `ExperimentGrid`, `call_experiment`, `setup_logger_kwargs` |
|
||||
| `spinup.utils.plot` | The plotter (documented in ch6) |
|
||||
|
||||
## Anti-patterns
|
||||
- **Calling `mpi_avg_grads` after `optimizer.step()`.** The averaging must happen before the
|
||||
adaptive optimizer consumes the gradients, or workers diverge in optimizer state.
|
||||
- **Skipping `setup_pytorch_for_mpi()`.** The thread-clobbering problem it fixes is a real
|
||||
performance failure, not a theoretical one.
|
||||
- **Expecting `log_tabular` to be idempotent.** It wipes the accumulated state; calling it
|
||||
twice for the same key in one epoch gives you statistics over nothing.
|
||||
- **Reaching for `mpi_tf` as a general distributed-training solution.** The docs say to use
|
||||
horovod instead if you need something serious.
|
||||
|
||||
## Key Takeaways
|
||||
1. `store` / `log_tabular` / `dump_tabular` is the whole logger interface, and the state
|
||||
resets on `log_tabular`.
|
||||
2. `EpochLogger` aggregates across MPI workers as well as across an epoch.
|
||||
3. MPI PyTorch is three calls in a fixed order: setup, sync params, average grads before step.
|
||||
4. The logger is also the model-saving and config-recording path (ch5).
|
||||
5. `mpi_tf` is acknowledged as hacky; horovod is the recommended alternative.
|
||||
|
||||
## Connects To
|
||||
- **Ch 4**: ExperimentGrid usage and the CLI it backs.
|
||||
- **Ch 5**: the outputs the logger writes — `progress.txt`, `config.json`, the save directories.
|
||||
- **Ch 6**: the plotter, which reads what `dump_tabular` writes.
|
||||
- **Ch 2**: OpenMPI, the system dependency all of this rests on.
|
||||
|
|
@ -0,0 +1,87 @@
|
|||
# Cheatsheet
|
||||
|
||||
Decision rules, thresholds and tells from Spinning Up. One page; keep it beside you.
|
||||
|
||||
## Which algorithm?
|
||||
|
||||
| If | Then | Because |
|
||||
|----|------|---------|
|
||||
| Discrete actions | PPO (or VPG to learn) | DDPG, TD3 and shipped SAC are continuous-only |
|
||||
| Continuous, stability matters most | PPO | Directly optimizes performance; on-policy is stable |
|
||||
| Continuous, sample efficiency matters most | SAC, else TD3 | Off-policy data reuse; both are research-grade here |
|
||||
| You are learning the field | VPG first, then DQN, A2C, PPO, DDPG | Simplest first; complexity added gradually |
|
||||
| You need a research baseline for TRPO/PPO | OpenAI Baselines, not Spinning Up | Spinning Up's on-policy trio omits observation normalization and normalized value targets |
|
||||
| You need a research baseline for DDPG/TD3/SAC | Spinning Up is fine | Roughly at parity with best reported results |
|
||||
| Partial observability or pixels | None of these as shipped | All six are non-recurrent MLP actor-critics |
|
||||
|
||||
## Debugging a run that does not learn
|
||||
|
||||
1. **Assume it is a bug, not the hyperparameters.** Bad hyperparameters degrade performance;
|
||||
if yours resemble the paper's, they are probably not the cause.
|
||||
2. **Check tensor shapes.** A `[N]` vs `[N,1]` mismatch is broadcast-compatible, raises
|
||||
nothing, and silently corrupts the Bellman backup into an `[N,N]` matrix.
|
||||
3. **Check what your loss is computed on** — wrong equation, wrong distribution, or data
|
||||
piped to the wrong place are the three usual causes.
|
||||
4. **Do not read the policy-gradient loss as a health signal.** Only average return means
|
||||
anything; the loss can go to negative infinity while performance craters.
|
||||
5. **Instrument more.** Mean/std/min/max of returns, episode lengths and value estimates;
|
||||
objective losses; exploration parameters (policy entropy, epsilon). Watch videos too.
|
||||
6. **Test in more than one environment** once results look promising — code can work in one
|
||||
environment despite a breaking bug.
|
||||
|
||||
## Thresholds and defaults
|
||||
|
||||
| Quantity | Value | Source |
|
||||
|----------|-------|--------|
|
||||
| Debug-stage turnaround target | **under 5 minutes** locally | Ch 10 |
|
||||
| Random seeds, minimum | **3** | Ch 10 |
|
||||
| Random seeds, thorough | **10 or more** | Ch 10, Ch 13 |
|
||||
| Debug environments | CartPole-v0, InvertedPendulum-v0, FrozenLake-v0, HalfCheetah-v2 at 100-250 steps | Ch 10 |
|
||||
| From-scratch implementation size | ~250-300 lines; no-frills VPG ~80 | Ch 10 |
|
||||
| On-policy benchmark network | (64, 32), tanh | Ch 13 |
|
||||
| Off-policy benchmark network | (256, 256), relu | Ch 13 |
|
||||
| On-policy batch | 4000 env steps per update | Ch 13 |
|
||||
| Off-policy minibatch | 100 per gradient step | Ch 13 |
|
||||
| TD3 `policy_delay` | 2 | Ch 18 |
|
||||
| Polyak `rho` | in (0,1), usually close to 1 | Ch 17 |
|
||||
| Benchmark length | 3M timesteps, 5 MuJoCo envs | Ch 13 |
|
||||
| Plot smoothing used in the book | 11-epoch window | Ch 13 |
|
||||
|
||||
## Choosing a research frame
|
||||
|
||||
| Frame | Scope | Wraps up in | Main risk |
|
||||
|-------|-------|-------------|-----------|
|
||||
| Improve an existing approach | Narrow | A few months | Tweaks fail and you have no signal on what next |
|
||||
| Unsolved benchmark | Broad | Months to a year-plus | May need a breakthrough; long time, no progress |
|
||||
| New problem setting | Open-ended | Unbounded | You must design the benchmark too; cannot go looking for these |
|
||||
|
||||
## Rigor checklist before reporting
|
||||
|
||||
- [ ] Baseline tuned **as much as** your own method. Never handicap it.
|
||||
- [ ] All else held equal (e.g. comparable parameter counts across architecture variants).
|
||||
- [ ] At least 3 seeds, ideally 10 or more; plot mean and std dev.
|
||||
- [ ] Final runs launched fresh and **precommitted** — not the best or most interesting runs.
|
||||
- [ ] One ablation per design decision, so each claim is separately supported.
|
||||
- [ ] Checked the idea has not already been published.
|
||||
|
||||
## Tells and smells
|
||||
|
||||
- **A learning curve with no seed band** is telling you less than it looks like.
|
||||
- **`AverageEpRet` on an off-policy algorithm** is the noisy behavior policy, not performance.
|
||||
Use the plotter's `Performance` alias.
|
||||
- **Evaluating SAC without `--deterministic`** measures the wrong policy.
|
||||
- **A model-based agent that scores well but behaves badly** is exploiting model bias — the
|
||||
same shape as a policy exploiting Q-function errors.
|
||||
- **"Fits the Bellman equation well"** carries no guarantee of good policy performance.
|
||||
- **A single missing `squeeze`** in a critic is the archetype of silent failure.
|
||||
- **A paper's full trick list** is usually more than strictly necessary — try simplifications.
|
||||
- **A library abstraction** is good for reuse across algorithms and a cost when learning one.
|
||||
|
||||
## Quick commands
|
||||
|
||||
```bash
|
||||
python -m spinup.run ppo --env Walker2d-v2 --exp_name walker --seed 0 10 20
|
||||
python -m spinup.run test_policy data/walker/walker_s0 # add -d for SAC
|
||||
python -m spinup.run plot data/walker # prefix autocompletes
|
||||
python -m spinup.run [algo] --help # every kwarg is a flag
|
||||
```
|
||||
|
|
@ -0,0 +1,56 @@
|
|||
# Glossary
|
||||
|
||||
**Action space** — the set of all valid actions in an environment; discrete or continuous (Ch 7)
|
||||
**Action-value function** — see Q-function (Ch 7)
|
||||
**Advantage function** — `A^pi(s,a) = Q^pi(s,a) - V^pi(s)`; how much better a specific action is than randomly selecting one per pi (Ch 7, Ch 9)
|
||||
**Agent** — the learner and decider; often used interchangeably with "policy" (Ch 7)
|
||||
**Backtracking line search** — shrink the step by `alpha^j` until the exact KL constraint holds and surrogate advantage is positive (Ch 15)
|
||||
**Baseline** — any state-only function `b(s)` subtractable from the policy gradient weight without changing its expectation (Ch 9)
|
||||
**Bellman backup** — the right-hand side of a Bellman equation: reward plus next value (Ch 7)
|
||||
**Bellman equation** — self-consistency condition: the value of your starting point is the reward you expect from being there plus the value of where you land next (Ch 7)
|
||||
**Categorical policy** — stochastic policy for discrete actions, built like a classifier: logits then softmax (Ch 7)
|
||||
**Clipped double-Q learning** — learn two Q-functions and use the smaller value to form the target (Ch 18, Ch 19)
|
||||
**Deadly triad** — function approximation + bootstrapping + off-policy data, jointly causing instability in value learning (Ch 8)
|
||||
**Diagonal Gaussian policy** — stochastic policy for continuous actions; mean network plus a diagonal covariance represented as a vector of log stds (Ch 7)
|
||||
**Discount factor (gamma)** — in (0,1); makes an infinite reward sum converge and encodes "cash now beats cash later" (Ch 7)
|
||||
**EGLP lemma** — Expected Grad-Log-Prob: `E_{x~P_theta}[grad log P_theta(x)] = 0` (Ch 9)
|
||||
**Entropy** — `H(P) = E_{x~P}[-log P(x)]`; how random a random variable is (Ch 19)
|
||||
**Entropy regularization** — adding an entropy bonus to the reward at each timestep, scaled by `alpha` (Ch 19)
|
||||
**Epoch** — one experience-collection phase plus one policy update (Ch 9)
|
||||
**ExperimentGrid** — Spinning Up's hyperparameter ablation tool; backs the CLI (Ch 4, Ch 20)
|
||||
**Fully / partially observed** — whether the agent sees the complete state or only part of it (Ch 7)
|
||||
**GAE** — Generalized Advantage Estimation; used by Spinning Up's VPG, TRPO and PPO (Ch 9)
|
||||
**Log-derivative trick** — `grad P = P * grad log P` (Ch 9)
|
||||
**Markov property** — transitions depend only on the most recent state and action (Ch 7)
|
||||
**MDP** — Markov Decision Process, the 5-tuple `<S, A, R, P, rho_0>` (Ch 7)
|
||||
**Model** — a function predicting state transitions and rewards (Ch 8)
|
||||
**Model-based / model-free** — whether the agent has or learns a model (Ch 8)
|
||||
**MPC** — model-predictive control: plan over a window, execute the first action, discard the rest, re-plan (Ch 8)
|
||||
**MSBE** — Mean-Squared Bellman Error, the loss deep Q-learning minimizes (Ch 17)
|
||||
**Natural Policy Gradient** — TRPO's analytic solution before the line search is added (Ch 15)
|
||||
**Observation** — a partial description of a state, possibly omitting information (Ch 7)
|
||||
**Off-policy** — updates may use data collected at any point in training (Ch 3, Ch 8)
|
||||
**On-policy** — updates use only data from the most recent policy (Ch 3, Ch 8)
|
||||
**Optimal policy (pi\*)** — `argmax_pi J(pi)` (Ch 7)
|
||||
**Parameterized policy** — a policy whose output depends on adjustable parameters, e.g. network weights (Ch 7)
|
||||
**Performance** — the plotter's family-aware alias, resolving to `AverageEpRet` (on-policy) or `AverageTestEpRet` (off-policy) (Ch 6)
|
||||
**Policy** — the rule for choosing actions; deterministic `mu(s)` or stochastic `pi(.|s)` (Ch 7)
|
||||
**Policy delay** — TD3's ratio of Q-updates to policy updates; recommended 2 (Ch 18)
|
||||
**Policy gradient** — `grad_theta J(pi_theta)` (Ch 9)
|
||||
**Polyak averaging** — target update `phi_targ <- rho*phi_targ + (1-rho)*phi`; `polyak` in the code (Ch 17)
|
||||
**Q-function** — `Q^pi(s,a)`, expected return from taking a in s then following pi (Ch 7)
|
||||
**Replay buffer** — the stored set of past transitions that off-policy algorithms sample from (Ch 17)
|
||||
**Reparameterization trick** — sample by a deterministic function of state, parameters and independent noise (Ch 19)
|
||||
**Return** — cumulative reward over a trajectory; finite-horizon undiscounted or infinite-horizon discounted (Ch 7)
|
||||
**Reward-to-go** — `Rhat_t = sum_{t'=t}^T R(s_t', a_t', s_{t'+1})` (Ch 9)
|
||||
**Silent failure** — broken RL code that runs correctly and simply never learns (Ch 10, Ch 12)
|
||||
**Squashed Gaussian policy** — a Gaussian passed through `tanh` to bound actions; SAC's policy (Ch 19)
|
||||
**start_steps** — initial phase of uniform-random actions used by DDPG, TD3 and SAC (Ch 17)
|
||||
**State** — a complete description of the world, with nothing hidden (Ch 7)
|
||||
**Surrogate advantage** — importance-weighted advantage measuring new-policy performance on old-policy data (Ch 15)
|
||||
**Surrogate objective** — a local approximation of `J(pi_theta)` that is safe to maximize (Ch 8, Ch 16)
|
||||
**Target network** — a time-lagged copy of a network used to compute stable Bellman targets (Ch 17)
|
||||
**Target policy smoothing** — clipped noise added to the target action to prevent exploiting sharp Q peaks (Ch 18)
|
||||
**Trajectory (tau)** — a sequence of states and actions; also called episode or rollout (Ch 7)
|
||||
**Trust region** — a KL-divergence bound on how far the new policy may move from the old (Ch 15)
|
||||
**Value function** — `V^pi(s)`, expected return from s following pi (Ch 7)
|
||||
|
|
@ -0,0 +1,131 @@
|
|||
# Patterns
|
||||
|
||||
Concrete techniques from Spinning Up, each with when to use it, how it works, and what it costs.
|
||||
|
||||
## Reward-to-Go Weighting
|
||||
**When to use** — always, over full-trajectory-return weighting. It is a two-line change.
|
||||
**How** — weight each `grad log pi(a_t|s_t)` by `sum_{t'=t}^T r_t'` instead of `R(tau)`.
|
||||
**Trade-offs** — strictly reduces variance at no cost. The dropped terms had zero mean and
|
||||
nonzero variance, so removing them removes pure noise and cuts the sample trajectories needed.
|
||||
(Ch 9)
|
||||
|
||||
## Value Function Baseline
|
||||
**When to use** — any policy gradient method. Standard in VPG, TRPO, PPO and A2C.
|
||||
**How** — subtract `V_phi(s_t)` from the weight; fit `V_phi` by minimizing
|
||||
`E[(V_phi(s_t) - Rhat_t)^2]` with one or more gradient steps per epoch, starting from the
|
||||
previous parameters, so it tracks the most recent policy.
|
||||
**Trade-offs** — provably unbiased by EGLP; reduces variance and gives faster, more stable
|
||||
learning. Costs a second network and its fitting budget. Under-fitting it is one of the most
|
||||
drastic performance failures in policy gradients (Exercise 2.1). (Ch 9, Ch 12)
|
||||
|
||||
## Trust Region via KL Constraint
|
||||
**When to use** — when a single bad policy gradient step can collapse performance, and you
|
||||
can afford second-order machinery.
|
||||
**How** — maximize surrogate advantage subject to `Dbar_KL <= delta`; Taylor-expand, solve
|
||||
analytically via Lagrangian duality, then backtracking line search until the exact KL
|
||||
constraint holds and surrogate advantage is positive. Use conjugate gradient for `H^{-1}g`.
|
||||
**Trade-offs** — buys monotone-ish, fast improvement and larger safe steps. Costs complexity,
|
||||
Hessian-vector products, and (in Spinning Up) no PyTorch implementation. (Ch 15)
|
||||
|
||||
## Clipped Surrogate Objective
|
||||
**When to use** — as the first-order replacement for a trust region. The default modern choice.
|
||||
**How** — `L = min(ratio * A, g(eps, A))` where `g` caps at `(1+eps)A` for positive advantage
|
||||
and `(1-eps)A` for negative. Take multiple minibatch SGD steps per batch.
|
||||
**Trade-offs** — much simpler than TRPO and empirically at least as good, but it removes the
|
||||
*incentive* to move far rather than guaranteeing you do not. Pair it with KL early stopping.
|
||||
(Ch 16)
|
||||
|
||||
## KL Early Stopping
|
||||
**When to use** — alongside clipping, whenever you take many gradient steps per batch.
|
||||
**How** — if the mean KL-divergence of the new policy from the old exceeds a threshold, stop
|
||||
taking gradient steps for this batch.
|
||||
**Trade-offs** — one extra measurement per step; catches the drift the clip alone allows.
|
||||
Other PPO implementations use different tricks here. (Ch 16)
|
||||
|
||||
## Experience Replay Buffer
|
||||
**When to use** — every off-policy value-learning algorithm.
|
||||
**How** — store `(s,a,r,s',d)` tuples; sample minibatches uniformly for MSBE minimization.
|
||||
Licensed by the fact that the Bellman equation is indifferent to how transitions were collected.
|
||||
**Trade-offs** — wrong in both directions. Too small and you overfit to the most recent data
|
||||
and things break; too large and learning slows. It needs tuning. (Ch 17)
|
||||
|
||||
## Target Networks
|
||||
**When to use** — any MSBE minimization, because the target otherwise depends on the
|
||||
parameters being trained.
|
||||
**How** — keep a lagged copy. DQN-style: copy every fixed number of steps. DDPG-style:
|
||||
polyak average once per main update, `phi_targ <- rho*phi_targ + (1-rho)*phi`, `rho` near 1.
|
||||
**Trade-offs** — stabilizes learning; adds a network and a hyperparameter, and slows the
|
||||
propagation of new information into the target. (Ch 17)
|
||||
|
||||
## Amortizing a Continuous Argmax into a Policy
|
||||
**When to use** — an inner `max` over a continuous variable inside a loop you run constantly.
|
||||
**How** — assume differentiability with respect to that variable, learn `mu_theta(s)` by
|
||||
gradient ascent on `Q_phi(s, mu_theta(s))`, and substitute `max_a Q(s,a) ~= Q(s, mu(s))`.
|
||||
**Trade-offs** — turns an intractable per-step optimization into a forward pass. Costs an
|
||||
approximation whose errors the policy will actively exploit. (Ch 17)
|
||||
|
||||
## Clipped Double-Q
|
||||
**When to use** — whenever a learned Q-function is being maximized over by a policy, i.e.
|
||||
the whole DDPG family.
|
||||
**How** — learn two Q-functions, use `min` of the two target values as the shared regression
|
||||
target for both.
|
||||
**Trade-offs** — fends off overestimation, which is DDPG's dominant failure. Costs a second
|
||||
critic and introduces some underestimation bias. TD3 uses `Q_phi_1` alone in the policy loss;
|
||||
SAC uses the `min` there too. (Ch 18, Ch 19)
|
||||
|
||||
## Delayed Policy Updates
|
||||
**When to use** — when policy updates destabilize the Bellman target.
|
||||
**How** — update the policy and all target networks once per `policy_delay` critic updates;
|
||||
the paper recommends 2.
|
||||
**Trade-offs** — damps volatility; slows policy improvement per environment step. (Ch 18)
|
||||
|
||||
## Target Policy Smoothing
|
||||
**When to use** — deterministic policies over continuous actions.
|
||||
**How** — `a'(s') = clip(mu_targ(s') + clip(eps, -c, c), a_Low, a_High)`, `eps ~ N(0, sigma)`.
|
||||
**Trade-offs** — regularizes Q along the action dimension so a spurious sharp peak cannot be
|
||||
exploited. A stochastic policy (SAC) gets a similar effect for free. (Ch 18, Ch 19)
|
||||
|
||||
## Entropy Regularization
|
||||
**When to use** — when premature convergence to a bad local optimum is the risk, or when you
|
||||
want an explicit explore-exploit dial.
|
||||
**How** — add `alpha * H(pi(.|s_t))` to the reward at each timestep; propagate the term into
|
||||
the value functions and the Bellman target.
|
||||
**Trade-offs** — higher `alpha` means more exploration and faster later learning; the right
|
||||
value is environment-specific and needs careful tuning. The entropy-constrained variant that
|
||||
adapts `alpha` is generally preferred by practitioners over the fixed one. (Ch 19)
|
||||
|
||||
## Reparameterization Trick for Stochastic Policies
|
||||
**When to use** — differentiating through an expectation whose distribution depends on the
|
||||
parameters.
|
||||
**How** — sample by a deterministic function of state, parameters and independent noise:
|
||||
`a~ = tanh(mu_theta(s) + sigma_theta(s) * xi)`, `xi ~ N(0,I)`, converting the expectation over
|
||||
actions into an expectation over noise.
|
||||
**Trade-offs** — makes the policy differentiable end to end. The `tanh` squash bounds actions
|
||||
but changes the distribution; log-probabilities are still closed-form. (Ch 19)
|
||||
|
||||
## Uniform Random Warm-Up (start_steps)
|
||||
**When to use** — every off-policy algorithm here, at the beginning of training.
|
||||
**How** — for a fixed number of steps take actions sampled uniformly over valid actions, then
|
||||
switch to normal exploration.
|
||||
**Trade-offs** — fills the replay buffer with diverse data before the policy can bias it.
|
||||
Wasted steps if the environment is expensive. (Ch 17, Ch 18, Ch 19)
|
||||
|
||||
## Multi-Seed Reporting with Variance Bands
|
||||
**When to use** — every reported RL result.
|
||||
**How** — at least 3 seeds, 10 or more to be thorough; plot the mean as a solid line and the
|
||||
standard deviation as a shaded band; smooth for display only.
|
||||
**Trade-offs** — multiplies compute by the seed count. Without it, deep RL's seed sensitivity
|
||||
means two seed groups can look like different distributions entirely. (Ch 10, Ch 13)
|
||||
|
||||
## Precommitted Final Runs
|
||||
**When to use** — before reporting any comparison.
|
||||
**How** — use the tuning stage to form hypotheses, then launch fresh final experiments for
|
||||
every method compared and commit in advance to reporting whatever comes out.
|
||||
**Trade-offs** — a weak form of preregistration; costs one more full experiment round and
|
||||
removes the largest source of accidental self-deception. (Ch 10)
|
||||
|
||||
## Per-Claim Ablation
|
||||
**When to use** — any method with more than one design decision, which is all of them.
|
||||
**How** — swap out or remove each design element separately and measure.
|
||||
**Trade-offs** — turns one bundled claim into several separately-confident ones. Costs one
|
||||
experiment per element. Also the fastest way to learn which parts you can delete. (Ch 10)
|
||||
|
|
@ -1,6 +1,6 @@
|
|||
site_name: Claude Code Skills & Agent Plugins
|
||||
site_url: https://alirezarezvani.github.io/claude-skills/
|
||||
site_description: "380 production-ready skills across 20 domains, 96 marketplace plugins, and 138 slash commands — engineering, product, marketing, compliance, finance, research, and agent tooling. Works with Claude Code, OpenAI Codex, Gemini CLI, Cursor, Hermes Agent, Mistral Vibe, OpenClaw, and 6 more AI coding tools. Open source, MIT licensed, zero dependencies."
|
||||
site_description: "388 production-ready skills across 20 domains, 99 marketplace plugins, and 150 slash commands — engineering, product, marketing, compliance, finance, research, and agent tooling. Works with Claude Code, OpenAI Codex, Gemini CLI, Cursor, Hermes Agent, Mistral Vibe, OpenClaw, and 6 more AI coding tools. Open source, MIT licensed, zero dependencies."
|
||||
site_author: Alireza Rezvani
|
||||
repo_url: https://github.com/alirezarezvani/claude-skills
|
||||
repo_name: alirezarezvani/claude-skills
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue