mirror of
https://github.com/alirezarezvani/claude-skills.git
synced 2026-08-28 04:24:58 +00:00
feat(engineering): compile OpenAI's Spinning Up in Deep RL into a knowledge-base plugin
Runs engineering/book-to-skill end to end on its first real source: OpenAI's Spinning Up in Deep RL (MIT, (c) 2018 OpenAI; primarily developed by Joshua Achiam). Cloned openai/spinningup and compiled its docs/ reStructuredText tree (38 files, ~37k words, ~49K tokens) through the full pipeline -- extract --mode technical, analysis, 20 chapter files, glossary/patterns/cheatsheet, master SKILL.md, validator, plugin emitter. The compiled skill passes book_skill_validator.py in --strict mode with every file inside budget: a 2,101-token resident core (cap 4,000) plus 20 on-demand chapters averaging ~1,256 tokens each. Chapter structure follows the source's own toctree rather than a heading scan: user documentation (ch01-06), Introduction to RL Parts 1-3 (ch07-09), the researcher essay / key papers / exercises / benchmarks (ch10-13), one chapter per algorithm in lineage order (ch14-19: VPG to TRPO to PPO, DDPG to TD3 and SAC), and the logger/MPI/ExperimentGrid utilities (ch20). Rights basis is open-license, not fair use -- the emitter's Step-11 gate refuses a shareable package without one. Upstream's MIT notice is reproduced in full in the plugin's LICENSE beside this package's own, and README.md names the source, the author and the source's frozen version; a sidecar JSON is not a license notice. Also fixes a defect the emitter only reveals at its final step: skill_plugin_emitter.py wrote its whole `source` provenance block into plugin.json, on a stale inline claim that `source`/`attribution` were approved extension fields. Claude Code rejects an entire manifest on any unrecognized key (issue #954) and scripts/check_plugin_json.py hard-fails such a manifest, so every package the emitter produced failed the blocking CI gate on commit. _plugin_manifest() now emits spec fields only and a new _authoring_notes() writes .claude-plugin/authoring-notes.json. Recorded as deviation 26 in engineering/book-to-skill/README.md; the printed marketplace.json snippet is unchanged, since `source` is a valid key there. Counters: skills 386 -> 387, agents 116 -> 117, commands 146 -> 147, plugins 97 -> 98. Tools and references unchanged -- a compiled knowledge base ships notes, not scripts. All blocking CI gates verified locally: compileall, check_plugin_json --all, check_skill_names, check_paths, check_frontmatter, check_dual_publish, check_model_freshness, smoke_scripts (692/692), derive_counters --check. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UySnyf5upm4y8xhYA3w6yw
This commit is contained in:
Alireza Rezvani
parent
f2bac0a8f2
commit
800a0d5672
37 changed files with 3062 additions and 42 deletions
|
|
@ -8,7 +8,7 @@
|
|||
"homepage": "https://github.com/alirezarezvani/claude-skills",
|
||||
"repository": "https://github.com/alirezarezvani/claude-skills",
|
||||
"metadata": {
|
||||
"description": "386 production-ready skills across 20 domains (engineering, engineering-core, marketing, product, c-level, c-level-agents, compliance-os, project management, RA/QM, business growth, finance, productivity, marketing top-level, research, research-ops, business-operations, commercial, markdown-html, loop-library, plus standards). 723 Python tools, 838 reference guides, 116 agents (cs-* + personas), 146 slash commands across 97 marketplace plugins. v2.11.2 vendors engineering/skillopt-sleep — a verbatim copy of microsoft/SkillOpt's stdlib-only skillopt_sleep engine + Claude Code plugin surface, giving a local agent a nightly gated self-improvement cycle (read-only session harvest -> mine -> offline replay -> held-out-gated CLAUDE.md/SKILL.md edits -> staged for explicit /skillopt-sleep adopt). productivity/fable-goal (unreleased, post-v2.11.1) converts a rambling description of a desired outcome into one polished /goal prompt for a fresh autonomous session. v2.11.1 turns product-team and project-management into agent-harness domains: fork-orchestrators with deterministic goal routers, a Jira MCP snapshot bridge (Kanban flow metrics + Monte Carlo forecasting), a delegation-governance loop gate, a continuous-discovery cadence tracker, and an Opportunity Solution Tree linter, with /cs:pm and /cs:product command families. v2.10.3 completes the markdown-html domain with md-slides — slide-deck converter (arrow-key / Space / PgDn / Home/End / P keyboard navigation + presenter mode with split-view clock + speaker notes + next-slide preview + URL-hash deep linking like #3 for direct slide jumps + @media print page-per-slide for browser-native PDF export). Reuses md-document's markdown parser; vanilla JS only (no framework runtime); Prism.js opt-in via --syntax. Joins md-review (v2.10.2 code-review converter), md-document (v2.10.1 long-form converter), and the v2.10.0 foundation (orchestrator + design-system). Compatible with Claude Code, Codex CLI, Gemini CLI, Cursor, OpenClaw, Hermes Agent, Mistral Vibe, and 5 more coding agents.",
|
||||
"description": "387 production-ready skills across 20 domains (engineering, engineering-core, marketing, product, c-level, c-level-agents, compliance-os, project management, RA/QM, business growth, finance, productivity, marketing top-level, research, research-ops, business-operations, commercial, markdown-html, loop-library, plus standards). 723 Python tools, 838 reference guides, 117 agents (cs-* + personas), 147 slash commands across 98 marketplace plugins. v2.11.2 vendors engineering/skillopt-sleep — a verbatim copy of microsoft/SkillOpt's stdlib-only skillopt_sleep engine + Claude Code plugin surface, giving a local agent a nightly gated self-improvement cycle (read-only session harvest -> mine -> offline replay -> held-out-gated CLAUDE.md/SKILL.md edits -> staged for explicit /skillopt-sleep adopt). productivity/fable-goal (unreleased, post-v2.11.1) converts a rambling description of a desired outcome into one polished /goal prompt for a fresh autonomous session. v2.11.1 turns product-team and project-management into agent-harness domains: fork-orchestrators with deterministic goal routers, a Jira MCP snapshot bridge (Kanban flow metrics + Monte Carlo forecasting), a delegation-governance loop gate, a continuous-discovery cadence tracker, and an Opportunity Solution Tree linter, with /cs:pm and /cs:product command families. v2.10.3 completes the markdown-html domain with md-slides — slide-deck converter (arrow-key / Space / PgDn / Home/End / P keyboard navigation + presenter mode with split-view clock + speaker notes + next-slide preview + URL-hash deep linking like #3 for direct slide jumps + @media print page-per-slide for browser-native PDF export). Reuses md-document's markdown parser; vanilla JS only (no framework runtime); Prism.js opt-in via --syntax. Joins md-review (v2.10.2 code-review converter), md-document (v2.10.1 long-form converter), and the v2.10.0 foundation (orchestrator + design-system). Compatible with Claude Code, Codex CLI, Gemini CLI, Cursor, OpenClaw, Hermes Agent, Mistral Vibe, and 5 more coding agents.",
|
||||
"version": "2.12.0"
|
||||
},
|
||||
"plugins": [
|
||||
|
|
@ -2050,6 +2050,28 @@
|
|||
"engineering"
|
||||
],
|
||||
"category": "development"
|
||||
},
|
||||
{
|
||||
"name": "spinning-up-deep-rl",
|
||||
"source": "./engineering/spinning-up-deep-rl",
|
||||
"description": "Knowledge base compiled from OpenAI's Spinning Up in Deep RL (MIT, Joshua Achiam) by engineering/book-to-skill. A resident core carries the RL optimization problem, the model-free taxonomy, the policy-gradient template with its five valid weights, the safe-step family (VPG to TRPO to PPO) and the overestimation family (DDPG to TD3 and SAC); 20 on-demand chapters cover key concepts and MDPs, the algorithm taxonomy and model bias, the policy gradient derivation with the log-derivative trick and EGLP lemma, Achiam's researcher essay, the key-papers topic map, the exercises including the silent DDPG broadcasting bug, the benchmark parity disclosure, one chapter per algorithm, and the logger/MPI/ExperimentGrid utilities. Ships a glossary, a patterns file with 16 techniques, a decision cheatsheet with thresholds, a cs-spinning-up-deep-rl agent and /cs:spinning-up-deep-rl. Structured study notes, not a reproduction of the source.",
|
||||
"version": "1.0.0",
|
||||
"author": {
|
||||
"name": "Alireza Rezvani"
|
||||
},
|
||||
"keywords": [
|
||||
"knowledge-base",
|
||||
"book-to-skill",
|
||||
"reinforcement-learning",
|
||||
"deep-rl",
|
||||
"policy-gradient",
|
||||
"ppo",
|
||||
"sac",
|
||||
"td3",
|
||||
"openai-spinning-up",
|
||||
"engineering"
|
||||
],
|
||||
"category": "development"
|
||||
}
|
||||
]
|
||||
}
|
||||
|
|
|
|||
40
CHANGELOG.md
40
CHANGELOG.md
|
|
@ -7,6 +7,46 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|||
|
||||
## [Unreleased]
|
||||
|
||||
### Added — engineering/spinning-up-deep-rl: the first book compiled by book-to-skill
|
||||
|
||||
Knowledge-base plugin compiled end-to-end by `engineering/book-to-skill` from OpenAI's
|
||||
[Spinning Up in Deep RL](https://spinningup.openai.com/) (MIT, Copyright (c) 2018 OpenAI;
|
||||
primarily developed by Joshua Achiam). 20 chapters, a glossary, a patterns file and a
|
||||
decision cheatsheet, behind a 2,101-token resident core.
|
||||
|
||||
- **The full pipeline, not a hand-write.** `openai/spinningup` cloned, its `docs/`
|
||||
reStructuredText tree (38 files, ~37k words, ~49K tokens) run through
|
||||
`extract_document.py --mode technical` → analysis → chapter files → supporting files →
|
||||
master `SKILL.md` → `book_skill_validator.py` → `skill_plugin_emitter.py`. The validator
|
||||
passes clean in `--strict` mode and every file is inside budget.
|
||||
- **Rights basis `open-license`, stated and honoured.** The emitter's Step-11 gate refuses a
|
||||
shareable package without one. MIT permits derivative distribution; upstream's notice is
|
||||
reproduced in full in the plugin's `LICENSE` beside this package's own, and `README.md`
|
||||
names the source, the author and the source's frozen version.
|
||||
- **Structure follows the source's own `toctree`.** User documentation (ch01-06), Introduction
|
||||
to RL Parts 1-3 (ch07-09), resources — the researcher essay, key papers, exercises,
|
||||
benchmarks (ch10-13), one chapter per algorithm in lineage order (ch14-19: VPG → TRPO → PPO,
|
||||
DDPG → TD3 → SAC), and the logger / MPI / ExperimentGrid utilities (ch20).
|
||||
- **The cheatsheet carries the judgment a glossary cannot** — the under-5-minute debug
|
||||
turnaround, the 3-seed minimum (10+ to be thorough), family-specific benchmark network
|
||||
defaults, and Spinning Up's own parity disclosure: DDPG/TD3/SAC are research-grade,
|
||||
VPG/TRPO/PPO are not, and the docs say to use OpenAI Baselines for those.
|
||||
- **Counters:** skills 386 → 387; agents 116 → 117; commands 146 → 147; plugins 97 → 98.
|
||||
Tools and references unchanged — a compiled knowledge base ships notes, not scripts.
|
||||
|
||||
### Fixed — book-to-skill's plugin emitter produced manifests this repo's CI rejects
|
||||
|
||||
`skill_plugin_emitter.py` wrote its whole `source` provenance block into `plugin.json`, with an
|
||||
inline comment asserting that `source` and `attribution` were approved extension fields. That had
|
||||
been true and no longer was: Claude Code rejects an entire manifest on any unrecognized key
|
||||
(issue #954), and `scripts/check_plugin_json.py` hard-fails such a manifest, pointing at
|
||||
`.claude-plugin/authoring-notes.json` instead. Every package the emitter produced therefore failed
|
||||
the blocking CI gate the moment it was committed — a defect at the very last step of the pipeline,
|
||||
which is why it had gone unnoticed. `_plugin_manifest()` now emits spec fields only and a new
|
||||
`_authoring_notes()` writes the sidecar. Recorded as deviation 26 in
|
||||
`engineering/book-to-skill/README.md`. The printed `marketplace.json` snippet is unchanged: `source`
|
||||
is a valid key there, which is how it leaked into the manifest originally.
|
||||
|
||||
### Added — marketing/linkedin: organic LinkedIn presence with the platform rules in code
|
||||
|
||||
New `marketing/linkedin/` plugin, answering
|
||||
|
|
|
|||
50
CLAUDE.md
50
CLAUDE.md
File diff suppressed because one or more lines are too long
12
README.md
12
README.md
|
|
@ -1,6 +1,6 @@
|
|||
# Claude Code Skills & Plugins — Agent Skills for Every Coding Tool
|
||||
|
||||
**386 production-ready Claude Code skills, plugins, and agent skills for 13 AI coding tools.**
|
||||
**387 production-ready Claude Code skills, plugins, and agent skills for 13 AI coding tools.**
|
||||
|
||||
The most comprehensive open-source library of Claude Code skills and agent plugins — also works with OpenAI Codex, Gemini CLI, Cursor, and 9 more coding agents. Reusable expertise packages covering engineering, DevOps, marketing (incl. AEO — Answer Engine Optimization for LLM citation), security (PreToolUse hooks), compliance, C-level advisory (incl. founder-mode CFO/CMO/CRO/CPO/COO/CHRO/CISO/GC/CDO/CAIO/CCO/VPE personas + 21 /cs:* slash commands), productivity (capture/email/reflect/weekly-review/deep-work/meetings), an academic research stack (litreview/grants/dossier/patent/syllabus/pulse/notebooklm/deep-research + hybrid router), and enterprise Research Operations (clinical-research/research-finance/market-research/product-research, v2.9.0).
|
||||
|
||||
|
|
@ -10,10 +10,10 @@ The most comprehensive open-source library of Claude Code skills and agent plugi
|
|||
[^vibe]: Mistral Vibe is also **BYO-sync tier**: the repo ships a pre-generated `.vibe/skills/claude-skills/` tree, run `./scripts/vibe-install.sh` once locally to install into `~/.vibe/skills/`. Same agentskills.io SKILL.md standard — no format conversion. Docs: <https://docs.mistral.ai/mistral-vibe/agents-skills>.
|
||||
|
||||
[](https://opensource.org/licenses/MIT)
|
||||
[](#skills-overview)
|
||||
[](#agents)
|
||||
[](#skills-overview)
|
||||
[](#agents)
|
||||
[](#personas)
|
||||
[](#commands)
|
||||
[](#commands)
|
||||
[](https://github.com/alirezarezvani/claude-skills/stargazers)
|
||||
[](https://getskillcheck.com)
|
||||
|
||||
|
|
@ -152,12 +152,12 @@ Run `./scripts/convert.sh --tool all` to generate tool-specific outputs locally.
|
|||
|
||||
## Skills Overview
|
||||
|
||||
**386 production-ready skills across 20 domains:**
|
||||
**387 production-ready skills across 20 domains:**
|
||||
|
||||
| Domain | Skills | Highlights | Details |
|
||||
|--------|--------|------------|---------|
|
||||
| **🔧 Engineering — Core** | 53 | Architecture, frontend, backend, fullstack, QA, DevOps, SecOps, AI/ML, data, Playwright Pro (test gen, flaky fix, migrations), self-improving agent (auto-memory curation), security suite, a11y audit, **named-persona-adversarial-review** (review via named engineering philosophies), **embedded-iot-mentor** (MCU/board selection, firmware-reuse-first, breadboard-MVP discipline) | [engineering-team/](engineering-team/) |
|
||||
| **⚡ Engineering — POWERFUL** | 91 | Agent designer, RAG architect, database designer, CI/CD builder, security auditor, MCP builder, AgentHub, Helm charts, Terraform, self-eval, llm-wiki, tc-tracker, autoresearch-agent, **reliability portfolio** (feature-flags-architect, kubernetes-operator, chaos-engineering, slo-architect), ship-gate, security-guidance PreToolUse hook, **Matt Pocock skills** (write-a-skill, caveman, grill-me, handoff, grill-with-docs), **zero-hallucination-coder** (Discuss→Map→Decompose→Execute→Verify), **agent-harness** (goal→plan→execute→verify→close loops over any domain), **memory-engineering** (price the memory write path, pick which cost to pay, audit FACT/SKILL/LOG density, gate on a forgetting policy), **skillopt-sleep** (nightly gated self-evolution from real Claude Code sessions, vendored from microsoft/SkillOpt), **book-to-skill** (compile a book, docs folder, or spec collection into a knowledge-base skill, then package it as a plugin), **boost-asio-pro** (async C++ networking — version-gated coroutine/callback styles, strand discipline), **human-gate** (batched human review as a structured artifact + a gate that refuses to close on open blockers), **agent-memory** (four-tier L0-L3 memory ladder over Claude Code hooks; promotion earned by recurrence across sessions and days, redacted or contested claims refuse to promote, nothing reaches CLAUDE.md without a human adopt), **skill-doctor** (grade the agent setup from real session history — rubric-judged, deterministically gated, evidence-traced skill edits) | [engineering/](engineering/) |
|
||||
| **⚡ Engineering — POWERFUL** | 92 | Agent designer, RAG architect, database designer, CI/CD builder, security auditor, MCP builder, AgentHub, Helm charts, Terraform, self-eval, llm-wiki, tc-tracker, autoresearch-agent, **reliability portfolio** (feature-flags-architect, kubernetes-operator, chaos-engineering, slo-architect), ship-gate, security-guidance PreToolUse hook, **Matt Pocock skills** (write-a-skill, caveman, grill-me, handoff, grill-with-docs), **zero-hallucination-coder** (Discuss→Map→Decompose→Execute→Verify), **agent-harness** (goal→plan→execute→verify→close loops over any domain), **memory-engineering** (price the memory write path, pick which cost to pay, audit FACT/SKILL/LOG density, gate on a forgetting policy), **skillopt-sleep** (nightly gated self-evolution from real Claude Code sessions, vendored from microsoft/SkillOpt), **book-to-skill** (compile a book, docs folder, or spec collection into a knowledge-base skill, then package it as a plugin), **boost-asio-pro** (async C++ networking — version-gated coroutine/callback styles, strand discipline), **human-gate** (batched human review as a structured artifact + a gate that refuses to close on open blockers), **agent-memory** (four-tier L0-L3 memory ladder over Claude Code hooks; promotion earned by recurrence across sessions and days, redacted or contested claims refuse to promote, nothing reaches CLAUDE.md without a human adopt), **skill-doctor** (grade the agent setup from real session history — rubric-judged, deterministically gated, evidence-traced skill edits), **spinning-up-deep-rl** (knowledge base compiled from OpenAI's Spinning Up in Deep RL — RL fundamentals, the model-free taxonomy, policy-gradient derivations, and one chapter per reference algorithm) | [engineering/](engineering/) |
|
||||
| **🎯 Product** | 17 | Product manager, agile PO, strategist, UX researcher, UI design, landing pages, SaaS scaffolder, analytics, experiment designer, discovery, roadmap communicator, code-to-prd, apple-hig-expert | [product-team/](product-team/) |
|
||||
| **📣 Marketing** | 49 | 8 pods: Content, SEO + AEO (`aeo` — E-E-A-T audit, citation tracking across 5 LLMs) + local (`local-seo-manager` — GBP/NAP/Map-Pack), CRO, Channels, Growth, Intelligence, Sales + `business-name-fit` (cross-cultural naming) + context foundation + orchestration router | [marketing-skill/](marketing-skill/) |
|
||||
| **🚀 Productivity** | 12 | `capture` (brain-dump-to-action), `email` pair (inbox-setup + inbox-triage), `reflect` (journal), `handoff` (Matt Pocock-inspired), `andreessen` (market-first decision mode), `roast` (5-angle idea panel → GO/RESHAPE/KILL), `fable-goal` (ramble → autonomous /goal prompt), `weekly-review` (GTD loop with refusal gate), `deep-work` (time-blocking + shallow-work budget), `meetings` (cost gate + agenda + action items), `swedish-mentor` (CEFR-leveled Swedish learning paths) | [productivity/](productivity/) |
|
||||
|
|
|
|||
|
|
@ -3,6 +3,6 @@
|
|||
"derived_from": "https://github.com/virgiliojr94/book-to-skill",
|
||||
"original_author": "virgiliojr94",
|
||||
"original_license": "MIT",
|
||||
"derivation_note": "The extraction library (book_to_skill/ — config, exceptions, sanitize, dependencies, utils, and the 7 per-format parsers) is vendored from upstream and carries upstream's multi-format chains, chapter detection across 5 script families, and Unicode/XXE hardening. 25 numbered deviations are recorded in README.md, which is the authoritative list: the workflow was rewritten Claude-Code-first for this repo's conventions, the 3 upstream tools were reworked into 4 argparse CLIs with --help/--sample, install-on-import behaviour was replaced with a report-only default, and Step 11 (plugin emission with a rights gate) plus the validator's budget and index families are additions with no upstream counterpart."
|
||||
"derivation_note": "The extraction library (book_to_skill/ — config, exceptions, sanitize, dependencies, utils, and the 7 per-format parsers) is vendored from upstream and carries upstream's multi-format chains, chapter detection across 5 script families, and Unicode/XXE hardening. 26 numbered deviations are recorded in README.md, which is the authoritative list: the workflow was rewritten Claude-Code-first for this repo's conventions, the 3 upstream tools were reworked into 4 argparse CLIs with --help/--sample, install-on-import behaviour was replaced with a report-only default, and Step 11 (plugin emission with a rights gate) plus the validator's budget and index families are additions with no upstream counterpart."
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -101,7 +101,7 @@ small; just read it."
|
|||
|
||||
## Deviations from upstream
|
||||
|
||||
**This numbered list is the authoritative record.** `plugin.json`'s
|
||||
**This numbered list is the authoritative record.** `.claude-plugin/authoring-notes.json`'s
|
||||
`attribution.derivation_note` summarizes it; if the two ever disagree, this list wins.
|
||||
|
||||
**Structural**
|
||||
|
|
@ -245,9 +245,10 @@ small; just read it."
|
|||
preventing hand-edit mistakes contradicted the manifest beside it); the symlink guard is
|
||||
backed by a **post-copy re-walk** that deletes the package if a link appeared during the
|
||||
copy, closing the check-then-act window rather than only narrowing it; and the manifest
|
||||
carries `source.license_scope` stating that the top-level `license` covers the package
|
||||
emits `source.license_scope` stating that the top-level `license` covers the package
|
||||
scaffolding, not the compiled notes — a distinction that previously lived only in README
|
||||
prose where a tool reading the manifest alone would miss it.
|
||||
prose where a tool reading the metadata alone would miss it. (Deviation 26 later moved
|
||||
that whole `source` block out of `plugin.json` and into the sidecar.)
|
||||
|
||||
21. **The documented quick-start actually runs.** SKILL.md's copy-paste block referenced
|
||||
`$WORKDIR` and `$SKILLS_HOME` without ever defining them — following it literally produced
|
||||
|
|
@ -297,6 +298,21 @@ small; just read it."
|
|||
leaves the victim's contents intact and is replaced by a 0600 file we own. Degrades to the
|
||||
previous path-based checks on platforms without `dir_fd`/`O_NOFOLLOW` (Windows).
|
||||
|
||||
26. **Provenance moved out of `plugin.json` into the sidecar the repo actually allows.**
|
||||
The manifest builder wrote its whole `source` block (spec, build pattern, source document,
|
||||
chapter count, distribution, `license_scope`, `rights_basis`) into `plugin.json`, and an
|
||||
inline comment asserted that `source` and `attribution` were "approved extension fields."
|
||||
That had been true and no longer was: Claude Code rejects an entire manifest on any
|
||||
unrecognized key (issue #954), and this repo's own `scripts/check_plugin_json.py` hard-fails
|
||||
a `plugin.json` carrying either field, pointing at `.claude-plugin/authoring-notes.json`
|
||||
instead. So every package the emitter produced failed the repo's blocking CI gate the moment
|
||||
it was committed — a defect that only surfaces at the very last step of the pipeline, which
|
||||
is why it survived. `_plugin_manifest()` now emits spec fields only, and a new
|
||||
`_authoring_notes()` writes the `source` block to `.claude-plugin/authoring-notes.json`.
|
||||
Note that `source` remains a *valid* key in a `marketplace.json` `plugins[]` entry, which is
|
||||
how it leaked into the manifest in the first place; the printed marketplace snippet is
|
||||
unchanged and still correct.
|
||||
|
||||
---
|
||||
|
||||
## Security audit
|
||||
|
|
|
|||
|
|
@ -166,13 +166,13 @@ def _assert_no_symlinks(skill_dir: Path) -> None:
|
|||
|
||||
|
||||
def _plugin_manifest(identity: dict, *, domain: str, author: str, author_url: str,
|
||||
repository: str, distribution: str, rights: str | None,
|
||||
source_note: str) -> dict:
|
||||
# Key set and the two approved extension fields (`source`, `attribution`) are
|
||||
# fixed by scripts/check_plugin_json.py. Anything else — including a
|
||||
# top-level "private" flag — fails the repo's CI gate, so the distribution
|
||||
# marker lives inside `source`.
|
||||
manifest = {
|
||||
repository: str) -> dict:
|
||||
# Spec fields ONLY. Claude Code's manifest validator rejects the whole
|
||||
# plugin.json on any unrecognized key (issue #954), and the repo's own gate
|
||||
# (scripts/check_plugin_json.py) hard-fails a manifest carrying `source` or
|
||||
# `attribution`. Provenance therefore goes to the sidecar file the validator
|
||||
# never reads — see _authoring_notes below.
|
||||
return {
|
||||
"name": identity["slug"],
|
||||
"description": identity["description"],
|
||||
"version": "1.0.0",
|
||||
|
|
@ -185,31 +185,42 @@ def _plugin_manifest(identity: dict, *, domain: str, author: str, author_url: st
|
|||
# the manifest sees the distinction that README.md makes in prose.
|
||||
"license": "MIT",
|
||||
"skills": [f"./skills/{identity['slug']}"],
|
||||
"source": {
|
||||
"spec": "generated by engineering/book-to-skill",
|
||||
"build_pattern": "book-to-skill conversion: extract -> analyze -> chapter files "
|
||||
"+ glossary + patterns + cheatsheet -> master SKILL.md",
|
||||
"distinct_from": "not a hand-authored skill; every claim traces to the source "
|
||||
"document named below",
|
||||
"source_document": source_note or identity["title"],
|
||||
"chapters": identity["chapters"],
|
||||
"distribution": distribution,
|
||||
"license_scope": (
|
||||
"The top-level `license` covers this package's scaffolding only. The "
|
||||
"compiled notes under skills/ are derived from the source document and "
|
||||
"carry that work's terms; see source.rights_basis."
|
||||
),
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def _authoring_notes(identity: dict, *, distribution: str, rights: str | None,
|
||||
source_note: str) -> dict:
|
||||
"""Provenance for `.claude-plugin/authoring-notes.json`.
|
||||
|
||||
The repo's schema allows exactly two keys here, `source` and `attribution`;
|
||||
a generated skill's provenance is a `source` block. This file is authoring
|
||||
metadata that Claude Code's manifest validator never reads, which is
|
||||
precisely why the fields live here rather than in plugin.json.
|
||||
"""
|
||||
source = {
|
||||
"spec": "generated by engineering/book-to-skill",
|
||||
"build_pattern": "book-to-skill conversion: extract -> analyze -> chapter files "
|
||||
"+ glossary + patterns + cheatsheet -> master SKILL.md",
|
||||
"distinct_from": "not a hand-authored skill; every claim traces to the source "
|
||||
"document named below",
|
||||
"source_document": source_note or identity["title"],
|
||||
"chapters": identity["chapters"],
|
||||
"distribution": distribution,
|
||||
"license_scope": (
|
||||
"plugin.json's top-level `license` covers this package's scaffolding only. "
|
||||
"The compiled notes under skills/ are derived from the source document and "
|
||||
"carry that work's terms; see source.rights_basis."
|
||||
),
|
||||
}
|
||||
if rights:
|
||||
manifest["source"]["rights_basis"] = rights
|
||||
manifest["source"]["rights_note"] = RIGHTS_BASES[rights]
|
||||
source["rights_basis"] = rights
|
||||
source["rights_note"] = RIGHTS_BASES[rights]
|
||||
if distribution == "local":
|
||||
# A local package is study notes from a work the user owns. The flag is
|
||||
# advisory metadata, not enforcement — it exists so a later publish step
|
||||
# (or a reviewer) can see the package was never cleared for sharing.
|
||||
manifest["source"]["cleared_for_distribution"] = False
|
||||
return manifest
|
||||
source["cleared_for_distribution"] = False
|
||||
return {"source": source}
|
||||
|
||||
|
||||
def _agent_markdown(identity: dict, domain: str, source_note: str) -> str:
|
||||
|
|
@ -407,10 +418,12 @@ def emit(*, skill_dir: Path, dest_root: Path, domain: str, author: str, author_u
|
|||
_assert_replaceable(package_root, resolved_dest, skill_dir)
|
||||
|
||||
manifest = _plugin_manifest(identity, domain=domain, author=author, author_url=author_url,
|
||||
repository=repository, distribution=distribution, rights=rights,
|
||||
source_note=source_note)
|
||||
repository=repository)
|
||||
notes = _authoring_notes(identity, distribution=distribution, rights=rights,
|
||||
source_note=source_note)
|
||||
files = {
|
||||
".claude-plugin/plugin.json": json.dumps(manifest, indent=2, ensure_ascii=False) + "\n",
|
||||
".claude-plugin/authoring-notes.json": json.dumps(notes, indent=2, ensure_ascii=False) + "\n",
|
||||
"README.md": _readme_markdown(identity, domain=domain, distribution=distribution,
|
||||
rights=rights, source_note=source_note),
|
||||
f"agents/cs-{identity['slug']}.md": _agent_markdown(identity, domain, source_note),
|
||||
|
|
|
|||
|
|
@ -0,0 +1,13 @@
|
|||
{
|
||||
"source": {
|
||||
"spec": "generated by engineering/book-to-skill",
|
||||
"build_pattern": "book-to-skill conversion: extract -> analyze -> chapter files + glossary + patterns + cheatsheet -> master SKILL.md",
|
||||
"distinct_from": "not a hand-authored skill; every claim traces to the source document named below",
|
||||
"source_document": "Spinning Up in Deep RL by Joshua Achiam (OpenAI)",
|
||||
"chapters": 20,
|
||||
"distribution": "shareable",
|
||||
"license_scope": "plugin.json's top-level `license` covers this package's scaffolding only. The compiled notes under skills/ are derived from the source document and carry that work's terms; see source.rights_basis.",
|
||||
"rights_basis": "open-license",
|
||||
"rights_note": "the source work carries a licence permitting derivative distribution"
|
||||
}
|
||||
}
|
||||
15
engineering/spinning-up-deep-rl/.claude-plugin/plugin.json
Normal file
15
engineering/spinning-up-deep-rl/.claude-plugin/plugin.json
Normal file
|
|
@ -0,0 +1,15 @@
|
|||
{
|
||||
"name": "spinning-up-deep-rl",
|
||||
"description": "Knowledge base from \"Spinning Up in Deep RL\" by Joshua Achiam (OpenAI, MIT-licensed). Use when applying Achiam's frameworks for RL fundamentals and MDPs, the model-free algorithm taxonomy, policy gradient derivations, the six reference algorithms (VPG, TRPO, PPO, DDPG, TD3, SAC), debugging silently-failing RL code, or running rigorous multi-seed RL experiments.",
|
||||
"version": "1.0.0",
|
||||
"author": {
|
||||
"name": "Alireza Rezvani",
|
||||
"url": "https://github.com/alirezarezvani"
|
||||
},
|
||||
"homepage": "https://github.com/alirezarezvani/claude-skills/tree/main/engineering/spinning-up-deep-rl",
|
||||
"repository": "https://github.com/alirezarezvani/claude-skills",
|
||||
"license": "MIT",
|
||||
"skills": [
|
||||
"./skills/spinning-up-deep-rl"
|
||||
]
|
||||
}
|
||||
58
engineering/spinning-up-deep-rl/LICENSE
Normal file
58
engineering/spinning-up-deep-rl/LICENSE
Normal file
|
|
@ -0,0 +1,58 @@
|
|||
This package has two layers with two different terms.
|
||||
|
||||
------------------------------------------------------------------------------
|
||||
1. Package scaffolding — plugin manifest, agent, command, README
|
||||
------------------------------------------------------------------------------
|
||||
|
||||
MIT License
|
||||
|
||||
Copyright (c) 2026 Alireza Rezvani
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the "Software"), to deal
|
||||
in the Software without restriction, including without limitation the rights
|
||||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||
copies of the Software, and to permit persons to whom the Software is
|
||||
furnished to do so, subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in
|
||||
all copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
|
||||
THE SOFTWARE.
|
||||
|
||||
------------------------------------------------------------------------------
|
||||
2. Compiled notes under skills/spinning-up-deep-rl/
|
||||
------------------------------------------------------------------------------
|
||||
|
||||
Derived from OpenAI's "Spinning Up in Deep RL" documentation
|
||||
(https://spinningup.openai.com/ · https://github.com/openai/spinningup),
|
||||
primarily developed by Joshua Achiam. Distributed under the same MIT terms,
|
||||
whose notice is reproduced below in full as that licence requires.
|
||||
|
||||
The MIT License
|
||||
|
||||
Copyright (c) 2018 OpenAI (http://openai.com)
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the "Software"), to deal
|
||||
in the Software without restriction, including without limitation the rights
|
||||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||
copies of the Software, and to permit persons to whom the Software is
|
||||
furnished to do so, subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in
|
||||
all copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
|
||||
THE SOFTWARE.
|
||||
49
engineering/spinning-up-deep-rl/README.md
Normal file
49
engineering/spinning-up-deep-rl/README.md
Normal file
|
|
@ -0,0 +1,49 @@
|
|||
# Spinning Up in Deep RL
|
||||
|
||||
Knowledge-base plugin compiled from **Spinning Up in Deep RL by Joshua Achiam (OpenAI)** by
|
||||
[`engineering/book-to-skill`](../book-to-skill/). 20 chapters indexed.
|
||||
|
||||
## What is in here
|
||||
|
||||
| File | Contents |
|
||||
|------|----------|
|
||||
| `skills/spinning-up-deep-rl/SKILL.md` | Core frameworks, chapter index, topic index (resident, under 4k tokens) |
|
||||
| `skills/spinning-up-deep-rl/chapters/` | One summary per chapter — loaded on demand, never all at once |
|
||||
| `skills/spinning-up-deep-rl/glossary.md` | Every significant term, alphabetized, with its chapter |
|
||||
| `skills/spinning-up-deep-rl/patterns.md` | Techniques and design patterns with trade-offs |
|
||||
| `skills/spinning-up-deep-rl/cheatsheet.md` | Decision rules, thresholds and trade-off matrices |
|
||||
|
||||
## Use
|
||||
|
||||
```
|
||||
/cs:spinning-up-deep-rl # core frameworks + chapter index
|
||||
/cs:spinning-up-deep-rl <topic> # resolve via topic index, read one chapter
|
||||
/cs:spinning-up-deep-rl ch05 # read one chapter summary
|
||||
```
|
||||
|
||||
Or invoke the `cs-spinning-up-deep-rl` agent for a working session anchored to this source.
|
||||
|
||||
## Provenance and limits
|
||||
|
||||
**Source:** OpenAI's [Spinning Up in Deep RL](https://spinningup.openai.com/)
|
||||
([openai/spinningup](https://github.com/openai/spinningup)), primarily developed by
|
||||
**Joshua Achiam**. Compiled from the `docs/` reStructuredText tree at the January 2020
|
||||
PyTorch update.
|
||||
|
||||
**Rights basis:** `open-license`. The source is **MIT, Copyright (c) 2018 OpenAI**, which
|
||||
permits derivative distribution. The full upstream notice is reproduced in
|
||||
[`LICENSE`](LICENSE) alongside this package's own; the top-level `license` field in
|
||||
`plugin.json` covers the scaffolding only.
|
||||
|
||||
Generated, not hand-authored: every claim traces to the source document. It carries that source's
|
||||
blind spots, and it is a set of structured notes — **not a copy of the work and not a substitute
|
||||
for reading it**.
|
||||
|
||||
**What it does not cover:** DQN and the discrete-action value-learning family, recurrent or
|
||||
convolutional architectures, partially-observed settings, model-based implementations, and any
|
||||
deep RL work after early 2020. The six implementations documented are educational; `ch13` records
|
||||
which are research-grade (DDPG, TD3, SAC) and which are not (VPG, TRPO, PPO).
|
||||
|
||||
Distribution: `shareable`. Regenerate or extend with
|
||||
`python3 engineering/book-to-skill/skills/book-to-skill/scripts/extract_document.py`, then re-run
|
||||
`book_skill_validator.py` before loading the result.
|
||||
|
|
@ -0,0 +1,37 @@
|
|||
---
|
||||
name: cs-spinning-up-deep-rl
|
||||
description: Answers from the knowledge base compiled from Spinning Up in Deep RL by Joshua Achiam (OpenAI). Loads the master frameworks first and reads a single chapter file on demand rather than the whole source. Refuses to answer beyond what the source covers.
|
||||
skills: engineering/spinning-up-deep-rl/skills/spinning-up-deep-rl
|
||||
domain: engineering
|
||||
model: opus
|
||||
tools: [Read, Grep, Glob]
|
||||
---
|
||||
|
||||
# Spinning Up in Deep RL — Knowledge Agent
|
||||
|
||||
## Voice
|
||||
|
||||
**Opening:** "Which framework or chapter are you reaching for?"
|
||||
**Forcing question:** "Is this something the source actually covers, or are you asking me to
|
||||
extrapolate past it?"
|
||||
**Closing:** "That is the author's formulation, from ch<N>. Anything past it is my inference, not theirs."
|
||||
|
||||
## Purpose
|
||||
|
||||
Applies the frameworks compiled from **Spinning Up in Deep RL by Joshua Achiam (OpenAI)** (20 chapters
|
||||
indexed) while the user works. Answers with the author's exact naming, then cites the chapter.
|
||||
|
||||
## How it navigates
|
||||
|
||||
1. Read `skills/spinning-up-deep-rl/SKILL.md` — Core Frameworks and both indexes.
|
||||
2. Match the question against the Topic Index; read **only** the chapter files it points to.
|
||||
3. Reach for `glossary.md` for a term, `patterns.md` for a technique, `cheatsheet.md` for a decision.
|
||||
4. Never load every chapter — that is the cost this skill exists to avoid.
|
||||
|
||||
## Hard rules
|
||||
|
||||
- **Cite the chapter.** Every framework claim names the chapter it came from.
|
||||
- **Do not extrapolate silently.** If the source does not cover it, say so before answering from
|
||||
general knowledge, and label which is which.
|
||||
- **Preserve exact naming.** The author's term is the interface; a paraphrase breaks lookup.
|
||||
- **Do not reproduce the source at length.** These are structured notes, not a copy of the work.
|
||||
|
|
@ -0,0 +1,30 @@
|
|||
---
|
||||
name: "cs-spinning-up-deep-rl"
|
||||
description: "/cs:spinning-up-deep-rl [topic | framework | chNN] — query the knowledge base compiled from Spinning Up in Deep RL by Joshua Achiam (OpenAI). Use when applying its frameworks while working, looking up a term, or reading one chapter's summary."
|
||||
---
|
||||
|
||||
# /cs:spinning-up-deep-rl — Spinning Up in Deep RL
|
||||
|
||||
**Command:** `/cs:spinning-up-deep-rl [topic | framework name | chNN]`
|
||||
|
||||
## When to run
|
||||
|
||||
- Applying a framework from this source to work in progress
|
||||
- Looking up the author's exact formulation of a term
|
||||
- Reading one chapter's compiled summary without opening the source
|
||||
- Checking whether the source covers a question at all
|
||||
|
||||
## What it does
|
||||
|
||||
1. Loads `engineering/spinning-up-deep-rl/skills/spinning-up-deep-rl/SKILL.md` — Core Frameworks plus the Chapter and Topic indexes.
|
||||
2. **No argument** → reports the core frameworks and the chapter index.
|
||||
3. **A topic or framework name** → resolves it through the Topic Index and reads only the
|
||||
matching chapter file.
|
||||
4. **`chNN`** → reads that chapter's summary directly.
|
||||
5. Answers with the author's naming and cites the chapter.
|
||||
|
||||
## Boundary
|
||||
|
||||
This command answers from **one source** (20 chapters indexed). Anything it does not
|
||||
cover gets said out loud rather than filled in — and hands-on work in your codebase belongs to the
|
||||
engineering skills, not here.
|
||||
|
|
@ -0,0 +1,192 @@
|
|||
---
|
||||
name: spinning-up-deep-rl
|
||||
description: "Knowledge base from \"Spinning Up in Deep RL\" by Joshua Achiam (OpenAI, MIT-licensed). Use when applying Achiam's frameworks for RL fundamentals and MDPs, the model-free algorithm taxonomy, policy gradient derivations, the six reference algorithms (VPG, TRPO, PPO, DDPG, TD3, SAC), debugging silently-failing RL code, or running rigorous multi-seed RL experiments."
|
||||
---
|
||||
|
||||
# Spinning Up in Deep RL
|
||||
|
||||
**Author**: Joshua Achiam (OpenAI) | **Source**: spinningup.readthedocs.io, MIT | **Chapters**: 20 | **Generated**: 2026-08-25
|
||||
|
||||
## How to Use This Skill
|
||||
|
||||
- **No argument** — load the core frameworks below
|
||||
- **A topic** — ask about `advantage function`, `target networks`, `entropy regularization`;
|
||||
I resolve it through the Topic Index and read that chapter file
|
||||
- **`chNN`** — I load that chapter's summary
|
||||
- **"what chapters do you have?"** — the full index
|
||||
|
||||
```
|
||||
/cs:spinning-up-deep-rl # core frameworks + chapter index
|
||||
/cs:spinning-up-deep-rl entropy regularization # topic index -> ch19, read that chapter
|
||||
/cs:spinning-up-deep-rl ch09 # one chapter summary
|
||||
```
|
||||
|
||||
When you ask about something not in Core Frameworks, I read the relevant chapter file before
|
||||
answering rather than guessing from the index.
|
||||
|
||||
---
|
||||
|
||||
## Core Frameworks & Mental Models
|
||||
|
||||
### The RL problem
|
||||
`pi* = argmax_pi J(pi)`, where `J(pi) = E_{tau~pi}[R(tau)]`. Every algorithm approximates
|
||||
this; where it substitutes a different objective (a Bellman residual, a surrogate), that
|
||||
substitution is the source of its failure modes. Four value functions — `V^pi`, `Q^pi`, `V*`,
|
||||
`Q*` — all obey Bellman self-consistency, and `a*(s) = argmax_a Q*(s,a)` is why Q-learning
|
||||
is a viable family at all. Advantage `A^pi(s,a) = Q^pi(s,a) - V^pi(s)` is the relative-quality
|
||||
signal policy gradients run on. (ch07)
|
||||
|
||||
### The two branching questions
|
||||
Place any algorithm by asking: **does it have or learn a model**, and **what does it learn**
|
||||
(policy, Q-function, value function, model). That generates the whole landscape. (ch08)
|
||||
|
||||
### Policy optimization vs Q-learning — the central trade-off
|
||||
- **Policy optimization** is *principled: you directly optimize the thing you want.* Stable
|
||||
and reliable. On-policy, so it cannot reuse data, so it is sample-hungry.
|
||||
- **Q-learning** only *indirectly* optimizes performance, by training `Q_theta` to satisfy a
|
||||
self-consistency equation. Many failure modes, so less stable. But substantially more
|
||||
sample efficient when it works, because it reuses everything.
|
||||
- **Satisfying the Bellman equations well carries no guarantee of good policy performance.**
|
||||
- The two are not exclusive — DDPG and SAC live between them deliberately. (ch08)
|
||||
|
||||
### The policy gradient template
|
||||
`grad J = E[ sum_t grad log pi_theta(a_t|s_t) * Phi_t ]`. Five valid choices of `Phi_t`:
|
||||
full return, reward-to-go, reward-to-go minus a baseline, `Q^pi`, and `A^pi`. All share an
|
||||
expectation and differ in variance. Two rules get you from the first to the last:
|
||||
- **Don't let the past distract you** — drop rewards obtained before the action. Those terms
|
||||
had zero mean and nonzero variance: pure noise.
|
||||
- **Baselines** — by the EGLP lemma, any state-only `b(s)` can be added or subtracted freely.
|
||||
The standard choice is `V^pi(s_t)`, learned by MSE regression onto reward-to-go. (ch09)
|
||||
|
||||
### The policy-gradient loss is not a loss function
|
||||
Its data distribution depends on the parameters, and it does not measure performance even in
|
||||
expectation. Only at the current parameters, with data from those parameters, does it have
|
||||
the negative gradient of performance. **You can send it to negative infinity while performance
|
||||
craters, and it usually will. Only average return means anything.** (ch09)
|
||||
|
||||
### Broken RL code almost always fails silently
|
||||
It runs fine; the agent just never learns. Usually something is computed with the wrong
|
||||
equation, on the wrong distribution, or piped to the wrong place. **If it doesn't work, assume
|
||||
there's a bug** before touching hyperparameters. Debug by measuring everything and reading the
|
||||
code critically. The archetype is one missing `squeeze`: a `[N]` vs `[N,1]` shape mismatch is
|
||||
broadcast-compatible, raises nothing, and silently turns the Bellman backup into an `[N,N]`
|
||||
matrix. (ch10, ch12)
|
||||
|
||||
### Learn by doing
|
||||
Write your own implementations, **shortest correct version** of each, **simplest algorithms
|
||||
first**. VPG, DQN, A2C, PPO, DDPG, roughly in that order; ~250-300 lines each. Single-threaded
|
||||
before parallel. **Iterate fast in simple environments — under 5 minutes turnaround at the
|
||||
debug stage.** Do not attempt Atari or Humanoid before the toy task works. Read papers for
|
||||
their ablations and supplementary material, but **do not overfit to paper details** (the
|
||||
original DDPG's architecture, init scheme and batch norm are not strictly necessary) **or to
|
||||
existing implementations** (their abstractions serve reuse, not your single use case). (ch10)
|
||||
|
||||
### Rigor: four standards
|
||||
1. **Fair comparisons** — tune the baseline as hard as your method; never handicap it.
|
||||
2. **Remove stochasticity as a confounder** — at least 3 seeds, 10 or more to be thorough.
|
||||
Two seed groups can produce curves that look like different distributions.
|
||||
3. **High-integrity experiments** — launch fresh final runs and precommit to reporting them.
|
||||
Tuning produces hypotheses; final runs produce conclusions.
|
||||
4. **Check each claim separately** — ablate every design decision. (ch10)
|
||||
|
||||
### The safe-step family (on-policy)
|
||||
VPG takes an unconstrained gradient step, so a single bad step can collapse performance.
|
||||
**TRPO** constrains the step in **KL-divergence between policies, not distance in parameter
|
||||
space**, then backtracking-line-searches until the exact constraint holds. **PPO** drops the
|
||||
constraint and instead **clips the objective so the policy gains nothing by moving far**,
|
||||
which is first-order, far simpler, and empirically at least as good. (ch14, ch15, ch16)
|
||||
|
||||
### The overestimation family (off-policy)
|
||||
DDPG amortizes the intractable continuous `max_a Q(s,a)` into a learned policy:
|
||||
`max_a Q(s,a) ~= Q(s, mu(s))`. It needs a **replay buffer** (licensed because the Bellman
|
||||
equation is indifferent to how data was collected) and **target networks** (because the target
|
||||
otherwise depends on the parameters being trained). Its failure mode is **Q-value
|
||||
overestimation, which the policy actively exploits**. **TD3** answers with clipped double-Q,
|
||||
delayed policy updates and target policy smoothing. **SAC** adds **entropy regularization**,
|
||||
making the explore-exploit trade-off an explicit coefficient `alpha`. (ch17, ch18, ch19)
|
||||
|
||||
---
|
||||
|
||||
## Chapter Index
|
||||
|
||||
| # | Title | Key Frameworks |
|
||||
|---|-------|----------------|
|
||||
| [ch01](chapters/ch01-introduction.md) | Introduction | The missing middle step, Code Design Philosophy |
|
||||
| [ch02](chapters/ch02-installation.md) | Installation | Install-then-verify, MuJoCo optionality |
|
||||
| [ch03](chapters/ch03-algorithm-lineup.md) | Algorithms: What's Included and Why | The two lineages, on/off-policy trade-off, code template |
|
||||
| [ch04](chapters/ch04-running-experiments.md) | Running Experiments | One flag per kwarg, ExperimentGrid, save-dir suffixes |
|
||||
| [ch05](chapters/ch05-experiment-outputs.md) | Experiment Outputs | Tools not files, watch-then-measure |
|
||||
| [ch06](chapters/ch06-plotting-results.md) | Plotting Results | `Performance` alias, prefix autocompletion, seed averaging |
|
||||
| [ch07](chapters/ch07-key-concepts-in-rl.md) | Part 1: Key Concepts in RL | MDPs, four value functions, Bellman equations, advantage |
|
||||
| [ch08](chapters/ch08-kinds-of-rl-algorithms.md) | Part 2: Kinds of RL Algorithms | Taxonomy, model bias, policy-opt vs Q-learning |
|
||||
| [ch09](chapters/ch09-intro-to-policy-optimization.md) | Part 3: Intro to Policy Optimization | Log-derivative trick, EGLP lemma, reward-to-go, baselines |
|
||||
| [ch10](chapters/ch10-spinning-up-as-a-researcher.md) | Spinning Up as a Deep RL Researcher | Learn by doing, three idea frames, four rigor standards |
|
||||
| [ch11](chapters/ch11-key-papers-in-deep-rl.md) | Key Papers in Deep RL | 13-section topic map |
|
||||
| [ch12](chapters/ch12-exercises.md) | Exercises | Problem Set 1 and 2, the silent DDPG bug |
|
||||
| [ch13](chapters/ch13-benchmarks.md) | Benchmarks | The parity disclosure, family-specific metrics |
|
||||
| [ch14](chapters/ch14-vpg.md) | Vanilla Policy Gradient | The six-step loop |
|
||||
| [ch15](chapters/ch15-trpo.md) | Trust Region Policy Optimization | KL trust region, line search, conjugate gradient |
|
||||
| [ch16](chapters/ch16-ppo.md) | Proximal Policy Optimization | PPO-Clip, KL early stopping |
|
||||
| [ch17](chapters/ch17-ddpg.md) | Deep Deterministic Policy Gradient | MSBE, replay buffers, target networks, polyak |
|
||||
| [ch18](chapters/ch18-td3.md) | Twin Delayed DDPG | Clipped double-Q, delayed updates, target smoothing |
|
||||
| [ch19](chapters/ch19-sac.md) | Soft Actor-Critic | Entropy regularization, reparameterization, squashed Gaussian |
|
||||
| [ch20](chapters/ch20-logger-and-utilities.md) | Logger, MPI Tools and Run Utils | EpochLogger pattern, MPI PyTorch order |
|
||||
|
||||
## Topic Index
|
||||
|
||||
- **Advantage function** ch07, ch09, ch14
|
||||
- **Baselines** ch09
|
||||
- **Bellman equations** ch07, ch17
|
||||
- **Benchmarks / parity** ch13, ch01
|
||||
- **Clipped double-Q** ch18, ch19
|
||||
- **Continuous action spaces** ch07, ch17
|
||||
- **Debugging / silent failure** ch10, ch12
|
||||
- **DDPG** ch17, ch03, ch08
|
||||
- **Entropy regularization** ch19
|
||||
- **Exploration vs exploitation** ch14, ch17, ch19
|
||||
- **GAE** ch09, ch14
|
||||
- **Installation** ch02
|
||||
- **KL divergence / trust region** ch15, ch16
|
||||
- **Logging** ch20, ch05
|
||||
- **MDPs** ch07
|
||||
- **Model-based RL** ch08
|
||||
- **MPI / parallelization** ch20, ch02, ch04
|
||||
- **MSBE** ch17
|
||||
- **Off-policy** ch03, ch08, ch17
|
||||
- **On-policy** ch03, ch08, ch14
|
||||
- **Papers / literature** ch11, ch10
|
||||
- **Plotting** ch06, ch13
|
||||
- **Policies (categorical, Gaussian, squashed)** ch07, ch19
|
||||
- **Policy gradient derivation** ch09
|
||||
- **PPO** ch16, ch03
|
||||
- **Q-learning** ch08, ch07
|
||||
- **Replay buffer** ch17
|
||||
- **Reparameterization trick** ch19, ch10
|
||||
- **Research process / rigor** ch10, ch13
|
||||
- **Reward-to-go** ch09
|
||||
- **Running experiments** ch04, ch05
|
||||
- **SAC** ch19, ch03, ch08
|
||||
- **Seeds / variance** ch10, ch13, ch04
|
||||
- **Target networks / polyak** ch17, ch18
|
||||
- **TD3** ch18, ch12
|
||||
- **TRPO** ch15, ch03
|
||||
- **Value functions** ch07, ch09
|
||||
- **VPG** ch14, ch09
|
||||
|
||||
## Supporting Files
|
||||
|
||||
- [glossary.md](glossary.md) · [patterns.md](patterns.md) · [cheatsheet.md](cheatsheet.md)
|
||||
|
||||
## Scope & Limits
|
||||
|
||||
Covers the Spinning Up documentation only, as of the January 2020 PyTorch update. It does
|
||||
**not** cover: DQN and the discrete-action value-learning family (referenced, never
|
||||
implemented here), recurrent or convolutional architectures, partially-observed settings,
|
||||
model-based implementations, exploration/meta-RL/hierarchy beyond ch11's reading list, or any
|
||||
deep RL work after early 2020. The six implementations are educational; ch13 says which are
|
||||
research-grade. For topics beyond this source, I say so rather than improvising.
|
||||
|
||||
---
|
||||
|
||||
*Compiled from OpenAI's Spinning Up in Deep RL documentation (MIT, Copyright (c) 2018 OpenAI),
|
||||
primarily developed by Joshua Achiam. Structured study notes, not a reproduction of the source.*
|
||||
|
|
@ -0,0 +1,102 @@
|
|||
# Chapter 1: Introduction
|
||||
|
||||
## Core Idea
|
||||
Spinning Up exists to be the missing middle step between "I want to work on deep RL"
|
||||
and "I can implement and evaluate a deep RL algorithm" — the field has no standard
|
||||
textbook, papers omit the design details that decide whether code works, and production
|
||||
RL libraries hide the algorithm inside framework abstractions.
|
||||
|
||||
## Frameworks Introduced
|
||||
- **The missing middle step**: the gap Spinning Up fills, between high-level awareness
|
||||
(what topics exist, why they matter) and the ability to transmute an algorithm into code.
|
||||
- When to use: deciding whether a resource teaches you deep RL or merely describes it.
|
||||
- How: judge a resource on whether it closes the theory-to-code gap. Papers give theory
|
||||
and omit details; libraries give code that hides the algorithm. Neither closes it alone.
|
||||
|
||||
- **Code Design Philosophy** (the four rules the implementations obey):
|
||||
- **As simple as possible while still being reasonably good.** Not state of the art;
|
||||
good enough to reach roughly the intended performance.
|
||||
- **Highly consistent with each other**, so understanding one makes the next painless.
|
||||
- **Almost completely self-contained** — virtually no shared code between algorithms
|
||||
except logging, saving, loading and MPI utilities. You can study one algorithm without
|
||||
following a chain of dependencies.
|
||||
- **Patterned to come as close to pseudocode as possible**, minimizing the theory-code gap.
|
||||
- How: when reading or writing RL code, prefer the version you can read top to bottom
|
||||
over the version that reuses the most code. Reuse is an engineering virtue that is a
|
||||
pedagogical cost.
|
||||
|
||||
- **Minimize tricks and minimize differences**: the implementations deliberately omit
|
||||
tricks present in the original papers (e.g. the regularization terms in the original
|
||||
Soft Actor-Critic code, observation normalization in all algorithms) and deliberately
|
||||
remove gratuitous differences between similar algorithms (DDPG, TD3 and SAC all run
|
||||
gradient updates after fixed intervals of environment interaction, so they compare cleanly).
|
||||
- When to use: any time you are comparing two algorithms and want the comparison to be
|
||||
about the algorithms rather than about their implementation choices.
|
||||
|
||||
## Key Concepts
|
||||
- **Deep RL**: reinforcement learning (learning to solve tasks by trial and error) combined
|
||||
with deep learning.
|
||||
- **Reasonably good**: achieves roughly the intended performance but does not necessarily
|
||||
match the best reported results in the literature on every task.
|
||||
- **Maintenance mode**: Spinning Up's current status. Breaking bugs get repaired; no major
|
||||
new features are planned.
|
||||
- **The Rosetta Stone goal**: the motivation for the January 2020 PyTorch update — the same
|
||||
algorithm expressed in two neural network libraries so the algorithm is separable from the
|
||||
library.
|
||||
|
||||
## Mental Models
|
||||
- Think of an RL library's abstraction layer as a **tax on learning**: good for code reuse
|
||||
between algorithms, unnecessary if you are writing one algorithm for one use case.
|
||||
- Use "**can I read this algorithm without opening another file?**" as the test of whether
|
||||
an implementation is written to be learned from.
|
||||
- Think of the resource landscape as **two failure modes**: papers that obscure key design
|
||||
details, and public implementations that are hard to read. Spinning Up targets both.
|
||||
|
||||
## Anti-patterns
|
||||
- **Using Spinning Up's implementations for scientific benchmarking comparisons**: they are
|
||||
"reasonably good," not best-reported. See ch13 for which ones are at parity (DDPG, TD3, SAC)
|
||||
and which are not (VPG, TRPO, PPO). Use OpenAI Baselines for TRPO/PPO research comparisons.
|
||||
- **Assuming an educational implementation is a research implementation**: the omitted tricks
|
||||
(observation normalization, normalized value regression targets) are exactly what separates
|
||||
the two.
|
||||
- **Treating deep RL as engineering-only**: the material explicitly serves people from
|
||||
professions with no connection to engineering or computer science who nonetheless need to
|
||||
make informed decisions about the technology.
|
||||
|
||||
## Worked Example
|
||||
Support history, read as a record of what the community actually asked for after release:
|
||||
|
||||
| Date | Event |
|
||||
|------|-------|
|
||||
| Nov 8, 2018 | Initial release, followed by three weeks of high-bandwidth support |
|
||||
| April 2019 | Six-month internal review of community feedback |
|
||||
| Jan 2020 | The PyTorch update ships |
|
||||
| Future | No major updates planned |
|
||||
|
||||
The April 2019 review surfaced exactly three requests, in priority order:
|
||||
1. **Implementations in other neural network libraries** — enough people had written their
|
||||
own PyTorch ports (Fired Up, Spinning Up Basic, Torching Up) that a "Rosetta Stone for
|
||||
deep RL" became the top priority. This one shipped.
|
||||
2. **Open source RL environments** (e.g. PyBullet) for benchmarks, examples and exercises,
|
||||
to avoid the proprietary MuJoCo dependency. Did not ship.
|
||||
3. **More algorithms**, especially Deep Q-Networks. Did not ship.
|
||||
|
||||
The lesson for anyone building an educational resource: the community's top request was not
|
||||
more content, it was the same content expressed in the framework they already use.
|
||||
|
||||
## Key Takeaways
|
||||
1. The barrier to entry in deep RL is not intelligence or math; it is that theory and code
|
||||
are documented in separate places and neither is written to connect to the other.
|
||||
2. Self-contained beats DRY when the reader is trying to learn the algorithm.
|
||||
3. Consistency across implementations is a teaching feature: differences that remain are
|
||||
real algorithmic differences.
|
||||
4. "Reasonably good" is a deliberate, stated performance target — check ch13 before citing
|
||||
any Spinning Up number as a benchmark.
|
||||
5. Deep RL is central to AI safety work in OpenAI's framing; the resource exists partly as
|
||||
a recruiting and capability-building pipeline for that.
|
||||
|
||||
## Connects To
|
||||
- **Ch 3**: which algorithms were chosen and why, and the code template they all share.
|
||||
- **Ch 10**: the essay on becoming a researcher — the practice counterpart to this chapter's
|
||||
philosophy.
|
||||
- **Ch 13**: the benchmark numbers that qualify "reasonably good."
|
||||
|
|
@ -0,0 +1,88 @@
|
|||
# Chapter 2: Installation
|
||||
|
||||
## Core Idea
|
||||
Spinning Up needs Python 3, OpenAI Gym and OpenMPI, on Linux or OSX; MuJoCo is optional
|
||||
but is the de facto benchmarking standard, and everything in the book works without it on
|
||||
the free Classic Control and Box2D environments.
|
||||
|
||||
## Frameworks Introduced
|
||||
- **Install then immediately verify with a real training run**: the install is not confirmed
|
||||
by a successful `pip install`, it is confirmed by an agent visibly learning.
|
||||
- When to use: every fresh environment, before debugging anything else.
|
||||
- How: run the install test below, then watch the policy and plot the curve. Three
|
||||
commands, three different subsystems (training, rendering, plotting) verified.
|
||||
|
||||
## Key Concepts
|
||||
- **OpenMPI**: the message-passing library used for the parallelized (on-policy) algorithms.
|
||||
Installed with `apt-get install libopenmpi-dev` on Ubuntu, `brew install openmpi` on Mac.
|
||||
- **MuJoCo**: a proprietary physics engine; free to trial and free for full-time students,
|
||||
otherwise paid. The de facto standard for benchmarking continuous-control deep RL.
|
||||
- **mujoco-py**: the Python bindings that let Gym use MuJoCo.
|
||||
- **Classic Control / Box2D**: Gym environment families that are completely free and
|
||||
sufficient to get started.
|
||||
- **Editable install**: `pip install -e .` from the cloned repo, so edits to the algorithm
|
||||
files take effect without reinstalling — the intended workflow for a repo you are meant
|
||||
to read and modify.
|
||||
|
||||
## Code Examples
|
||||
Environment and dependencies:
|
||||
|
||||
```bash
|
||||
conda create -n spinningup python=3.6
|
||||
conda activate spinningup
|
||||
|
||||
# Ubuntu
|
||||
sudo apt-get update && sudo apt-get install libopenmpi-dev
|
||||
# Mac OS X (requires Homebrew)
|
||||
brew install openmpi
|
||||
|
||||
git clone https://github.com/openai/spinningup.git
|
||||
cd spinningup
|
||||
pip install -e .
|
||||
```
|
||||
|
||||
Verify the install (roughly 10 minutes; leave it running and keep reading):
|
||||
|
||||
```bash
|
||||
python -m spinup.run ppo --hid "[32,32]" --env LunarLander-v2 \
|
||||
--exp_name installtest --gamma 0.999
|
||||
python -m spinup.run test_policy data/installtest/installtest_s0
|
||||
python -m spinup.run plot data/installtest/installtest_s0
|
||||
```
|
||||
|
||||
Optional MuJoCo, after following the mujoco-py README and obtaining a license:
|
||||
|
||||
```bash
|
||||
pip install gym[mujoco,robotics]
|
||||
python -m spinup.run ppo --hid "[32,32]" --env Walker2d-v2 --exp_name mujocotest
|
||||
```
|
||||
|
||||
## Reference Tables
|
||||
|
||||
| Requirement | Status | Notes |
|
||||
|-------------|--------|-------|
|
||||
| Python 3 | Required | Anaconda recommended; the docs pin 3.6 |
|
||||
| OpenAI Gym | Required | Installed by `pip install -e .`, MuJoCo envs excluded |
|
||||
| OpenMPI | Required | Needed for the MPI-parallelized on-policy algorithms |
|
||||
| Linux / OSX | Required | Windows unsupported and untested; one community workaround exists |
|
||||
| MuJoCo | Optional | Proprietary, licensed; preferred because of benchmarking convention |
|
||||
|
||||
## Anti-patterns
|
||||
- **Skipping the install test** and then debugging an algorithm that was never installed
|
||||
correctly. The install test costs ten unattended minutes.
|
||||
- **Treating a MuJoCo licence as a prerequisite for learning RL**: Classic Control and Box2D
|
||||
are free and adequate for everything up to benchmarking.
|
||||
- **Fighting package management ad hoc**: the docs go out of their way to point at conda
|
||||
explainers, because "I just installed this thing but it says it's not found" is the
|
||||
predicted failure, not an unusual one.
|
||||
|
||||
## Key Takeaways
|
||||
1. Install is Python 3 + Gym + OpenMPI, plus optional MuJoCo; Linux or OSX only.
|
||||
2. `pip install -e .` (editable) because you are expected to modify the code.
|
||||
3. Verification means an actual short PPO run, a rendered policy and a plot.
|
||||
4. Not having MuJoCo blocks benchmarking, not learning.
|
||||
|
||||
## Connects To
|
||||
- **Ch 4**: `python -m spinup.run` is the entry point every later chapter uses.
|
||||
- **Ch 13**: the benchmarks that make MuJoCo the convention.
|
||||
- **Ch 20**: the MPI utilities that OpenMPI enables.
|
||||
|
|
@ -0,0 +1,101 @@
|
|||
# Chapter 3: Algorithms — What's Included and Why
|
||||
|
||||
## Core Idea
|
||||
The six algorithms (VPG, TRPO, PPO, DDPG, TD3, SAC) are not a survey; they are two
|
||||
lineages of ideas, each starting from a foundational algorithm and progressively fixing
|
||||
its defining weakness, culminating in PPO and SAC.
|
||||
|
||||
## Frameworks Introduced
|
||||
- **The two lineages**:
|
||||
- **On-policy line: VPG to TRPO to PPO.** Each step buys back sample efficiency without
|
||||
giving up the stability that comes from directly optimizing the objective you care about.
|
||||
- **Off-policy line: DDPG to TD3 and SAC.** Each step mitigates the brittleness that comes
|
||||
from optimizing a Bellman self-consistency condition instead of performance itself.
|
||||
- When to use: placing any new algorithm you read about. Ask which line it is on and which
|
||||
weakness of its predecessor it claims to fix.
|
||||
|
||||
- **The central trade-off the lineup exposes**: on-policy algorithms *directly optimize the
|
||||
objective you care about* — policy performance — and it works out mathematically that
|
||||
you need on-policy data to compute the updates, so they cannot reuse old data. Off-policy
|
||||
algorithms exploit the Bellman optimality equations, which hold for *any* transition data,
|
||||
so they reuse everything — but satisfying Bellman's equations well carries **no guarantee**
|
||||
of good policy performance.
|
||||
- How: choose the family by which risk you can afford. Stability with a sample budget, or
|
||||
sample efficiency with a tuning budget.
|
||||
|
||||
- **The standard two-file code template**: every implementation splits into an *algorithm
|
||||
file* (experience buffer class, then one function that runs the algorithm) and a *core
|
||||
file* (utilities, the actor-critic constructors, the MLP actor-critic).
|
||||
- When to use: reading any Spinning Up algorithm, or structuring your own.
|
||||
|
||||
## Key Concepts
|
||||
- **On-policy**: each update uses only data collected by the most recent version of the policy.
|
||||
- **Off-policy**: each update can use data collected at any point in training, however the
|
||||
agent was exploring at the time.
|
||||
- **MLP actor-critic**: all six use non-recurrent multi-layer-perceptron actor-critics, which
|
||||
makes them suitable for fully-observed, non-image-based environments (e.g. Gym MuJoCo) and
|
||||
unsuitable, as shipped, for partial observability or pixels.
|
||||
- **Sample efficiency**: how much environment interaction is needed to reach a performance level.
|
||||
- **Deterministic policy gradients**: the theory (2014) that made DDPG possible — much younger
|
||||
than the policy gradient theory behind VPG, whose core elements go back to the late 80s.
|
||||
|
||||
## Reference Tables
|
||||
|
||||
| Algorithm | Family | Policy | Action spaces | Parallel (MPI) | PyTorch | TF1 |
|
||||
|-----------|--------|--------|---------------|----------------|---------|-----|
|
||||
| VPG | On-policy | Stochastic | Discrete + continuous | Yes | Yes | Yes |
|
||||
| TRPO | On-policy | Stochastic | Discrete + continuous | Yes | No | Yes |
|
||||
| PPO | On-policy | Stochastic | Discrete + continuous | Yes | Yes | Yes |
|
||||
| DDPG | Off-policy | Deterministic | Continuous only | No | Yes | Yes |
|
||||
| TD3 | Off-policy | Deterministic | Continuous only | No | Yes | Yes |
|
||||
| SAC | Off-policy | Stochastic | Continuous only (as shipped) | No | Yes | Yes |
|
||||
|
||||
## Code Examples
|
||||
The PyTorch algorithm function, in order — the template every implementation follows:
|
||||
|
||||
```
|
||||
1) Logger setup
|
||||
2) Random seed setting
|
||||
3) Environment instantiation
|
||||
4) Build the actor-critic module via the `actor_critic` function passed in as an argument
|
||||
5) Instantiate the experience buffer
|
||||
6) Set up callable loss functions that also return algorithm-specific diagnostics
|
||||
7) Make PyTorch optimizers
|
||||
8) Set up model saving through the logger
|
||||
9) Set up an update function: one epoch of optimization, or one step of descent
|
||||
10) Main loop: a) run the agent in the environment
|
||||
b) periodically update parameters per the algorithm's main equations
|
||||
c) log key performance metrics and save the agent
|
||||
```
|
||||
|
||||
The TF1 version is the same shape with four graph-construction steps inserted (placeholders,
|
||||
actor-critic graph, loss/diagnostic graph, training ops) and a session step.
|
||||
|
||||
## Mental Models
|
||||
- Read the lineup as **"progressions of ideas from the recent history of the field"**, not as
|
||||
a menu. VPG predates deep RL entirely; DDPG's theory is from 2014.
|
||||
- Treat **PPO and SAC as the two defaults** — they are close to state of the art on
|
||||
reliability and sample efficiency among policy-learning algorithms.
|
||||
- Think of the `actor_critic` argument as the seam: the algorithm function is fixed, the
|
||||
network construction is injected. That is also where the ch12 silent bug lives.
|
||||
|
||||
## Anti-patterns
|
||||
- **Reaching for DDPG on a discrete action space.** DDPG, TD3 and the shipped SAC are
|
||||
continuous-only. The max over actions is why (see ch17).
|
||||
- **Expecting these to work on Atari or partially-observed tasks as shipped**: MLP,
|
||||
non-recurrent, non-image.
|
||||
- **Picking an algorithm by recency**: TD3 and SAC were published roughly concurrently; SAC
|
||||
is not a successor to TD3, it is a parallel branch that borrowed the clipped double-Q trick.
|
||||
|
||||
## Key Takeaways
|
||||
1. Two families, one trade-off: directly optimize performance (on-policy, stable, sample-hungry)
|
||||
or exploit Bellman (off-policy, sample-efficient, no performance guarantee).
|
||||
2. PPO and SAC are the practical end points of each line.
|
||||
3. Continuous-only for the whole off-policy line, as implemented here.
|
||||
4. Every implementation is two files and the same ten-step function; learn the template once.
|
||||
5. TRPO has no PyTorch implementation in Spinning Up.
|
||||
|
||||
## Connects To
|
||||
- **Ch 8**: the full taxonomy this lineup is a slice of.
|
||||
- **Ch 14-19**: one chapter per algorithm, in lineage order.
|
||||
- **Ch 12**: the exercises that make the on-policy/off-policy distinction concrete.
|
||||
|
|
@ -0,0 +1,133 @@
|
|||
# Chapter 4: Running Experiments
|
||||
|
||||
## Core Idea
|
||||
`spinup/run.py` turns every keyword argument of every algorithm into a command-line flag,
|
||||
and turns a list of values for a flag into a series of experiments — so hyperparameter
|
||||
sweeps and multi-seed runs are one command, not a script.
|
||||
|
||||
## Frameworks Introduced
|
||||
- **One flag per kwarg**: if `kwarg` is a valid keyword argument of an algorithm function,
|
||||
`--kwarg` sets it. `python -m spinup.run [algo] --help` prints the docstring readout.
|
||||
- How: `python -m spinup.run [algo name] [experiment flags]`.
|
||||
|
||||
- **Multiple values means multiple experiments**: providing more than one value for an
|
||||
argument launches one experiment per combination, **in series**. They do not run in
|
||||
parallel because a single experiment already soaks up enough resources that concurrency
|
||||
buys no speedup.
|
||||
- When to use: seed sweeps (the ch10 rigor requirement) and hyperparameter ablations.
|
||||
|
||||
- **ExperimentGrid**: the in-script equivalent, based on but simpler than rllab's
|
||||
VariantGenerator. `spinup.run` uses one under the hood.
|
||||
- How: `eg.add(param_name, values, shorthand, in_name)` then `eg.run(thunk, **run_kwargs)`.
|
||||
`in_name` forces a parameter into the experiment name even when it does not vary.
|
||||
- Difference from the CLI: no shortcut kwargs — you must write `ac_kwargs:hidden_sizes`,
|
||||
not `hid`.
|
||||
|
||||
- **Shorthand-driven save directories**: results land in
|
||||
`data_dir/[outer_prefix]exp_name[suffix]/[inner_prefix]exp_name[suffix]_s[seed]`.
|
||||
The suffix encodes only the hyperparameters that *differ across the launched experiments*,
|
||||
and never the seed — so runs that differ only by seed group into the same folder, which is
|
||||
exactly what the plotter needs to average over seeds.
|
||||
|
||||
## Key Concepts
|
||||
- **`--env` / `--env_name`**: a Gym environment name, converted internally into the `env_fn`
|
||||
callable every algorithm actually takes.
|
||||
- **`--hid` / `--act`**: shortcut flags for `ac_kwargs:hidden_sizes` and `ac_kwargs:activation`,
|
||||
valid for all current algorithms.
|
||||
- **`--cpu` / `--num_cpu`**: launch with this many MPI-connected processes; `auto` uses all
|
||||
available. Raises an error for algorithms that do not support parallelization.
|
||||
- **`--dt` / `--datestamp`**: put timestamps in the save directory names.
|
||||
- **User-supplied shorthand**: square brackets after a flag, e.g. `--hid[h]`, control the
|
||||
directory-name abbreviation. Without one, a shorthand is derived automatically
|
||||
(`clip_ratio` becomes `cli`).
|
||||
- **`eval()` passthrough**: flag values pass through `eval()` before use, so you can name
|
||||
functions and objects directly, e.g. `--act torch.nn.ELU`.
|
||||
- **Dict kwargs**: `--key:v1 value_1 --key:v2 value_2` instead of `--key dict(v1=..., v2=...)`.
|
||||
|
||||
## Code Examples
|
||||
The quickstart, with every flag doing something distinct:
|
||||
|
||||
```bash
|
||||
python -m spinup.run ppo --exp_name ppo_ant --env Ant-v2 --clip_ratio 0.1 0.2 \
|
||||
--hid[h] [32,32] [64,32] --act torch.nn.Tanh --seed 0 10 20 --dt \
|
||||
--data_dir path/to/data
|
||||
```
|
||||
|
||||
Choosing the backend explicitly:
|
||||
|
||||
```bash
|
||||
python -m spinup.run ppo_pytorch --env Walker2d-v2 --exp_name walker
|
||||
python -m spinup.run ppo_tf1 --env Walker2d-v2 --exp_name walker
|
||||
# bare `ppo` reads spinup/user_config.py for the default backend
|
||||
```
|
||||
|
||||
From a script:
|
||||
|
||||
```python
|
||||
from spinup import ppo_pytorch as ppo
|
||||
import gym
|
||||
env_fn = lambda: gym.make('LunarLander-v2')
|
||||
ac_kwargs = dict(hidden_sizes=[64, 64])
|
||||
logger_kwargs = dict(output_dir='path/to/output_dir', exp_name='experiment_name')
|
||||
ppo(env_fn=env_fn, ac_kwargs=ac_kwargs, steps_per_epoch=5000, epochs=250,
|
||||
logger_kwargs=logger_kwargs)
|
||||
```
|
||||
|
||||
ExperimentGrid, from `spinup/examples/pytorch/bench_ppo_cartpole.py`:
|
||||
|
||||
```python
|
||||
from spinup.utils.run_utils import ExperimentGrid
|
||||
from spinup import ppo_pytorch
|
||||
import torch
|
||||
|
||||
eg = ExperimentGrid(name='ppo-pyt-bench')
|
||||
eg.add('env_name', 'CartPole-v0', '', True)
|
||||
eg.add('seed', [10*i for i in range(args.num_runs)])
|
||||
eg.add('epochs', 10)
|
||||
eg.add('steps_per_epoch', 4000)
|
||||
eg.add('ac_kwargs:hidden_sizes', [(32,), (64,64)], 'hid')
|
||||
eg.add('ac_kwargs:activation', [torch.nn.Tanh, torch.nn.ReLU], '')
|
||||
eg.run(ppo_pytorch, num_cpu=args.cpu)
|
||||
```
|
||||
|
||||
## Worked Example
|
||||
`python -m spinup.run ddpg_tf1 --env Hopper-v2 --hid[h] [300] [128,128] --act tf.nn.tanh tf.nn.relu`
|
||||
|
||||
Two `hid` values times two `act` values is four experiments, run in series, producing four
|
||||
suffixes:
|
||||
|
||||
```
|
||||
_h128-128_ac-actrelu
|
||||
_h128-128_ac-acttanh
|
||||
_h300_ac-actrelu
|
||||
_h300_ac-acttanh
|
||||
```
|
||||
|
||||
`h` came from the user-supplied `[h]`; `ac-act` was derived automatically from the true flag
|
||||
name `ac_kwargs:activation`. Add `--seed 0 10 20` and you get twelve runs in four folders —
|
||||
three seeds grouped per configuration, which is the grouping the plotter averages over.
|
||||
|
||||
## Anti-patterns
|
||||
- **Running the per-algorithm files directly** (`spinup/algos/BACKEND/ALGO/ALGO.py`). The
|
||||
command-line support there is vestigial, takes a different argument set, and is explicitly
|
||||
not the recommended way to run experiments.
|
||||
- **Using ZShell without escaping square brackets**: ZShell treats them as special characters
|
||||
and Spinning Up uses them for both list values and shorthands.
|
||||
- **One seed.** RL algorithms have high variance; the flag exists so you use it (ch10 asks
|
||||
for at least 3, ideally 10 or more).
|
||||
- **Expecting parallel experiment launches.** Sweeps run in series by design.
|
||||
|
||||
## Key Takeaways
|
||||
1. Every kwarg is a flag; every list of flag values is a sweep.
|
||||
2. Sweeps run in series; MPI parallelism is per-experiment via `--cpu`, and only for
|
||||
algorithms that support it.
|
||||
3. Save-directory suffixes encode only the varying hyperparameters, never the seed — that
|
||||
grouping is what makes multi-seed plotting work.
|
||||
4. `ExperimentGrid` is the scripted form and loses only the shortcut kwargs.
|
||||
5. Values go through `eval()`, so activations and other objects can be named on the command line.
|
||||
|
||||
## Connects To
|
||||
- **Ch 5**: what the directories those flags create actually contain.
|
||||
- **Ch 6**: the plotter, which consumes this directory structure and its autocompletion.
|
||||
- **Ch 10**: why multi-seed runs are a rigor requirement, not a convenience.
|
||||
- **Ch 20**: `ExperimentGrid`, `call_experiment` and `setup_logger_kwargs` in Run Utils.
|
||||
|
|
@ -0,0 +1,97 @@
|
|||
# Chapter 5: Experiment Outputs
|
||||
|
||||
## Core Idea
|
||||
Every run saves four things — a config record, a TSV of training metrics, the trained
|
||||
model, and a pickled copy of the environment — and only one of them (`config.json`) is
|
||||
ever meant to be read by hand.
|
||||
|
||||
## Frameworks Introduced
|
||||
- **Tools, not files**: `test_policy` loads from `pyt_save/` or `tf1_save/`, the plotter
|
||||
interprets `progress.txt`. Those are the correct interfaces. `config.json` is the only
|
||||
file you should ever open yourself, and only to remember what you ran.
|
||||
- When to use: any time you are tempted to parse a save directory manually.
|
||||
|
||||
- **Watch, then measure**: `test_policy` renders the agent so you can see behavior that no
|
||||
scalar metric would have shown you; `-nr` drops rendering when you only want the numbers.
|
||||
|
||||
## Key Concepts
|
||||
- **`progress.txt`**: tab-separated records of every metric the logger recorded, e.g. `Epoch`,
|
||||
`AverageEpRet`.
|
||||
- **`config.json`**: as-complete-as-possible dict of the args and kwargs used to launch the run.
|
||||
Record-keeping only — launching an experiment from a config file is not supported.
|
||||
- **`vars.pkl`**: algorithm state; currently used only to save a copy of the environment.
|
||||
- **`pyt_save/model.pt`**: a pickled PyTorch `nn.Module`; loading restores an ActorCritic
|
||||
object with an `act` method.
|
||||
- **`tf1_save/`**: `variables/`, `model_info.pkl` (key-to-tensor-name map) and `saved_model.pb`.
|
||||
Renamed from `simple_save/` on 2020-01-30.
|
||||
- **`DEFAULT_DATA_DIR`**: set in `spinup/user_config.py`; defaults to `spinningup/data`.
|
||||
|
||||
## Reference Tables
|
||||
|
||||
| File | Contents |
|
||||
|------|----------|
|
||||
| `pyt_save/` | PyTorch only. Everything needed to restore the agent and value functions |
|
||||
| `tf1_save/` | TF1 only. SavedModel plus the key-to-tensor map |
|
||||
| `config.json` | The launch args and kwargs. Non-serializable values become strings |
|
||||
| `progress.txt` | TSV of logged metrics across training |
|
||||
| `vars.pkl` | Pickled environment copy; may be empty if the env cannot be pickled |
|
||||
|
||||
`test_policy` flags:
|
||||
|
||||
| Flag | Default | Does |
|
||||
|------|---------|------|
|
||||
| `-l L`, `--len=L` | 0 | Max episode length; 0 means no maximum |
|
||||
| `-n N`, `--episodes=N` | 100 | Number of test episodes |
|
||||
| `-nr`, `--norender` | off | Print returns and lengths only; much faster |
|
||||
| `-i I`, `--itr=I` | -1 | Which saved snapshot; -1 is latest (see below) |
|
||||
| `-d`, `--deterministic` | off | **SAC only.** Use the deterministic mean policy |
|
||||
|
||||
## Code Examples
|
||||
```bash
|
||||
python -m spinup.run test_policy path/to/output_directory
|
||||
```
|
||||
|
||||
When the environment failed to pickle and `test_policy` raises `AssertionError: Environment
|
||||
not found!`, rebuild it by hand:
|
||||
|
||||
```python
|
||||
from spinup.utils.test_policy import load_policy_and_env, run_policy
|
||||
import your_env
|
||||
_, get_action = load_policy_and_env('/path/to/output_directory')
|
||||
env = your_env.make()
|
||||
run_policy(env, get_action)
|
||||
```
|
||||
|
||||
To keep snapshots from many points in training (off by default — algorithms overwrite the
|
||||
most recent one), change the line present in every algorithm:
|
||||
|
||||
```python
|
||||
logger.save_state({'env': env}, None) # default: one snapshot, overwritten
|
||||
logger.save_state({'env': env}, epoch) # keep per-epoch snapshots
|
||||
```
|
||||
|
||||
and then set `save_freq` to something reasonable — at the default of 1 you flood the output
|
||||
directory with one folder per epoch.
|
||||
|
||||
## Anti-patterns
|
||||
- **Forgetting `-d` when evaluating SAC.** SAC trains a stochastic policy but the correct
|
||||
evaluation policy is the deterministic mean. Without the flag you are measuring the wrong
|
||||
thing, and the flag is used for no other algorithm.
|
||||
- **Expecting to resume training.** Spinning Up implementations have no way to resume a
|
||||
partially-trained agent.
|
||||
- **Trying to relaunch from `config.json`.** It is a record, not an input.
|
||||
- **Assuming `vars.pkl` has your environment.** Gym Box2D environments in older Gym versions
|
||||
are known to fail to pickle, leaving it empty.
|
||||
|
||||
## Key Takeaways
|
||||
1. Four artifacts per run; use the tools for three of them and read only `config.json` by hand.
|
||||
2. SAC evaluation requires `--deterministic`; every other algorithm ignores it.
|
||||
3. Multi-snapshot saving is a two-line change plus a `save_freq` you must set deliberately.
|
||||
4. Environment pickling can silently fail — the recovery path is rebuilding the env in Python.
|
||||
5. No resume support; a killed run is a lost run.
|
||||
|
||||
## Connects To
|
||||
- **Ch 4**: the flags that create these directories.
|
||||
- **Ch 6**: `progress.txt` is what the plotter reads.
|
||||
- **Ch 19**: why SAC's evaluation policy differs from its training policy.
|
||||
- **Ch 20**: the EpochLogger that writes all of this.
|
||||
|
|
@ -0,0 +1,79 @@
|
|||
# Chapter 6: Plotting Results
|
||||
|
||||
## Core Idea
|
||||
The plotter reads a directory prefix, autocompletes every matching run beneath it, averages
|
||||
across seeds by default, and resolves the pseudo-metric `Performance` to the *correct*
|
||||
performance measure for each algorithm family — so a fair on-policy versus off-policy
|
||||
comparison is one command.
|
||||
|
||||
## Frameworks Introduced
|
||||
- **`Performance` as a family-aware alias**: `Performance` is not a real output of any
|
||||
algorithm. The plotter resolves it per logdir to `AverageEpRet` for on-policy algorithms
|
||||
and `AverageTestEpRet` for off-policy ones.
|
||||
- When to use: always, unless you specifically want one raw column. It is the default `-y`.
|
||||
- Why it matters: on-policy performance is the average return of the batch just collected;
|
||||
off-policy performance is measured by separate deterministic test rollouts. Plotting the
|
||||
same raw column for both would compare two different quantities.
|
||||
|
||||
- **Prefix autocompletion**: logdirs are searched recursively and prefixes expand. Give the
|
||||
plotter `data/bench_algo` and it finds `bench_algo1` and `bench_algo2` with all their seeds.
|
||||
- How: name experiments with a shared prefix at launch and comparison becomes free.
|
||||
|
||||
- **Average by default, `--count` to disaggregate**: by default, y-values are averaged across
|
||||
all results sharing an `exp_name` — typically identical experiments differing only in seed.
|
||||
`--count` shows each curve separately.
|
||||
|
||||
## Key Concepts
|
||||
- **`--xaxis` / `-x`**: which column is the x-axis. Default `TotalEnvInteracts`, i.e. sample
|
||||
efficiency is the default framing, not wall-clock or epochs.
|
||||
- **`--value` / `-y`**: which columns to graph; multiple values produce multiple graphs.
|
||||
- **`--smooth S`**: average over a fixed window of width S. The book's own benchmark plots
|
||||
use a window of 11 epochs (ch13).
|
||||
- **`--legend` / `-l`**: overrides the automatic legend, which uses `exp_name` from
|
||||
`config.json`. One string per *matched* directory, which may exceed the number of logdir
|
||||
arguments you passed.
|
||||
- **`--select` / `--exclude`**: keep or drop curves whose logdir contains all of / any of
|
||||
these substrings. The way to narrow an over-eager autocomplete.
|
||||
|
||||
## Code Examples
|
||||
```bash
|
||||
python -m spinup.run plot [path/to/output_directory ...] [--legend [LEGEND ...]] \
|
||||
[--xaxis XAXIS] [--value [VALUE ...]] [--count] [--smooth S] \
|
||||
[--select [SEL ...]] [--exclude [EXC ...]]
|
||||
```
|
||||
|
||||
Comparing two algorithms across all their seeds, relying on autocompletion:
|
||||
|
||||
```
|
||||
data/
|
||||
bench_algo1/
|
||||
bench_algo1-seed0/
|
||||
bench_algo1-seed10/
|
||||
bench_algo2/
|
||||
bench_algo2-seed0/
|
||||
bench_algo2-seed10/
|
||||
```
|
||||
|
||||
```bash
|
||||
python spinup/utils/plot.py data/bench_algo
|
||||
```
|
||||
|
||||
## Anti-patterns
|
||||
- **Passing one legend string per logdir argument.** Autocompletion may match several
|
||||
directories per argument; you need one legend entry per match, unless you narrowed the
|
||||
set with `--select` / `--exclude`.
|
||||
- **Plotting `AverageEpRet` for an off-policy algorithm** and calling it performance — that
|
||||
is the behavior policy with exploration noise, not the evaluated policy. Use `Performance`.
|
||||
- **Heavy smoothing to make a result look clean.** Smoothing is a display parameter; it does
|
||||
not change what the seeds did, and ch10 asks you to show seed variance rather than hide it.
|
||||
|
||||
## Key Takeaways
|
||||
1. `Performance` is the correct default because it is algorithm-family aware.
|
||||
2. Prefix autocompletion plus shared `exp_name` prefixes is the intended comparison workflow.
|
||||
3. Seed averaging is on by default; `--count` reveals the individual curves that average hides.
|
||||
4. The default x-axis is environment interactions — sample efficiency is the default question.
|
||||
|
||||
## Connects To
|
||||
- **Ch 5**: `progress.txt`, the plotter's input.
|
||||
- **Ch 4**: the suffix rules that group seeds into one folder for averaging.
|
||||
- **Ch 13**: the benchmark plots, produced with 10 seeds and an 11-epoch smoothing window.
|
||||
|
|
@ -0,0 +1,139 @@
|
|||
# Chapter 7: Part 1 — Key Concepts in RL
|
||||
|
||||
## Core Idea
|
||||
RL is the study of agents learning by trial and error; formally, the agent maximizes
|
||||
expected return J(pi) over trajectories, and almost every algorithm gets there through one
|
||||
of four value functions, all of which obey Bellman self-consistency equations.
|
||||
|
||||
## Frameworks Introduced
|
||||
- **The agent-environment interaction loop**: at every step the agent sees a (possibly
|
||||
partial) observation, chooses an action, and receives a reward. The environment changes
|
||||
because of the action, and may also change on its own.
|
||||
- How: name the five MDP pieces before writing any code — states S, actions A, reward
|
||||
function R, transition function P, start-state distribution rho_0.
|
||||
|
||||
- **The RL optimization problem**: `pi* = argmax_pi J(pi)`, where
|
||||
`J(pi) = E_{tau ~ pi}[R(tau)]` and the trajectory distribution is
|
||||
`P(tau|pi) = rho_0(s_0) * prod_t P(s_{t+1}|s_t,a_t) * pi(a_t|s_t)`.
|
||||
- When to use: as the definition every algorithm is trying to approximate. If an algorithm
|
||||
optimizes something else (a Bellman residual, a surrogate objective), that substitution
|
||||
is the source of its failure modes.
|
||||
|
||||
- **The four value functions**:
|
||||
- `V^pi(s)` — on-policy value: expected return starting in s and acting by pi forever.
|
||||
- `Q^pi(s,a)` — on-policy action-value: same, but take an arbitrary a first.
|
||||
- `V*(s)` — optimal value: expected return acting optimally from s.
|
||||
- `Q*(s,a)` — optimal action-value: take arbitrary a, then act optimally.
|
||||
- Two connections that come up constantly: `V^pi(s) = E_{a~pi}[Q^pi(s,a)]` and
|
||||
`V*(s) = max_a Q*(s,a)`.
|
||||
|
||||
- **The optimal-action shortcut**: if you have `Q*`, you get the optimal policy for free:
|
||||
`a*(s) = argmax_a Q*(s,a)`. There may be several maximizers, all optimal, but there is
|
||||
always an optimal policy that picks deterministically.
|
||||
- When to use: this is the entire justification for Q-learning as a family — learn `Q*`
|
||||
and the policy is a lookup. Ch17 explains what breaks when the argmax is over a
|
||||
continuous space.
|
||||
|
||||
- **Bellman equations**: "the value of your starting point is the reward you expect to get
|
||||
from being there, plus the value of wherever you land next."
|
||||
- On-policy: `V^pi(s) = E_{a~pi, s'~P}[r(s,a) + gamma V^pi(s')]`
|
||||
- Optimal: `V*(s) = max_a E_{s'~P}[r(s,a) + gamma V*(s')]`
|
||||
- The one crucial difference is the presence of the `max` over actions in the optimal form,
|
||||
reflecting that an agent free to choose must pick the highest-value action.
|
||||
- **Bellman backup** = the right-hand side, the reward-plus-next-value.
|
||||
|
||||
- **The advantage function**: `A^pi(s,a) = Q^pi(s,a) - V^pi(s)`. How much better taking a
|
||||
specific action is than randomly selecting one according to pi, assuming you follow pi
|
||||
afterwards.
|
||||
- When to use: whenever relative quality is what matters and absolute value is noise.
|
||||
Crucially important to policy gradient methods (ch9).
|
||||
|
||||
## Key Concepts
|
||||
- **State vs observation**: a state s is a complete description of the world; an observation
|
||||
o may omit information. Fully observed vs partially observed environments. Notation
|
||||
routinely writes s where o is technically correct.
|
||||
- **Action space**: the set of valid actions. **Discrete** (Atari, Go) versus **continuous**
|
||||
(robot control, real-valued vectors). The distinction has profound consequences: some
|
||||
algorithm families apply directly only to one case.
|
||||
- **Policy**: deterministic `a_t = mu(s_t)` or stochastic `a_t ~ pi(.|s_t)`. In deep RL these
|
||||
are **parameterized** — parameters theta or phi written as a subscript. "Policy" is often
|
||||
used interchangeably with "agent."
|
||||
- **Trajectory** (also **episode**, **rollout**): `tau = (s_0, a_0, s_1, a_1, ...)`.
|
||||
- **Finite-horizon undiscounted return**: `R(tau) = sum_{t=0}^{T} r_t`.
|
||||
- **Infinite-horizon discounted return**: `R(tau) = sum_{t=0}^{inf} gamma^t r_t`, with
|
||||
`gamma` in (0,1).
|
||||
- **MDP**: the 5-tuple `<S, A, R, P, rho_0>`. Markov property: transitions depend only on the
|
||||
most recent state and action, not on prior history.
|
||||
|
||||
## Code Examples
|
||||
A deterministic continuous-action policy is just an MLP:
|
||||
|
||||
```python
|
||||
pi_net = nn.Sequential(
|
||||
nn.Linear(obs_dim, 64), nn.Tanh(),
|
||||
nn.Linear(64, 64), nn.Tanh(),
|
||||
nn.Linear(64, act_dim)
|
||||
)
|
||||
obs_tensor = torch.as_tensor(obs, dtype=torch.float32)
|
||||
actions = pi_net(obs_tensor)
|
||||
```
|
||||
|
||||
**Categorical policies** (discrete actions) are built exactly like a classifier: observation
|
||||
in, layers, a final linear layer giving logits per action, softmax to probabilities. Sampling
|
||||
uses the framework's built-in categorical sampler. Log-likelihood is a vector index:
|
||||
`log pi_theta(a|s) = log [P_theta(s)]_a`.
|
||||
|
||||
**Diagonal Gaussian policies** (continuous actions) always have a network mapping observations
|
||||
to mean actions `mu_theta(s)`. The covariance is diagonal, so it is a vector, represented one
|
||||
of two ways:
|
||||
1. A single **state-independent** vector of log standard deviations — standalone parameters.
|
||||
*Spinning Up's VPG, TRPO and PPO do it this way.*
|
||||
2. A network `log sigma_theta(s)` mapping states to log standard deviations, optionally
|
||||
sharing layers with the mean network. *SAC does it this way, and SAC with state-independent
|
||||
log stds did not work (ch19).*
|
||||
|
||||
Log standard deviations, not standard deviations, because logs are free to range over
|
||||
(-inf, inf) while stds must be nonnegative, and unconstrained parameters are easier to train.
|
||||
Nothing is lost: exponentiate to recover.
|
||||
|
||||
Sampling: `a = mu_theta(s) + sigma_theta(s) * z` with `z ~ N(0, I)` (elementwise product).
|
||||
|
||||
Log-likelihood for a k-dimensional action:
|
||||
`log pi_theta(a|s) = -0.5 * ( sum_i [ (a_i - mu_i)^2 / sigma_i^2 + 2 log sigma_i ] + k log 2pi )`
|
||||
|
||||
## Mental Models
|
||||
- **A discount factor is two arguments in one**: intuitively, cash now beats cash later;
|
||||
mathematically, an infinite sum of rewards may not converge and is hard to work with.
|
||||
- **The formalism is starker than the practice.** Deep RL routinely sets up algorithms to
|
||||
optimize the *undiscounted* return while using discount factors when *estimating value
|
||||
functions*. Expect the line to be blurred in real code.
|
||||
- Treat **Q\* as a policy in disguise** and **V\* as a scoring function**: only the action-value
|
||||
form directly yields an action.
|
||||
- Ask of every value function: **is it time-dependent?** Unless stated otherwise, value
|
||||
functions mean infinite-horizon discounted return. Finite-horizon undiscounted value
|
||||
functions would need time as an argument — because what a state is worth depends on how
|
||||
much time is left.
|
||||
|
||||
## Anti-patterns
|
||||
- **Reading `s` in an equation as a true state.** In partially observed settings the action
|
||||
is conditioned on the observation; the notation is convention, not a claim.
|
||||
- **Assuming an algorithm ports across action-space types.** Moving a method between discrete
|
||||
and continuous spaces can require substantial rework, not a flag.
|
||||
- **Learning `V*` and expecting to act.** Without `Q*` or a model you cannot extract the
|
||||
action from the value.
|
||||
|
||||
## Key Takeaways
|
||||
1. The goal is always `argmax_pi E_{tau~pi}[R(tau)]`; everything else is machinery for
|
||||
approximating it.
|
||||
2. Four value functions, two of them optimal; `V^pi = E_a[Q^pi]` and `V* = max_a Q*` connect them.
|
||||
3. Bellman equations hold for all four; the `max` is what separates optimal from on-policy.
|
||||
4. `a*(s) = argmax_a Q*(s,a)` is why Q-learning is a viable family at all.
|
||||
5. Advantage `A = Q - V` is the relative-quality signal that policy gradients are built on.
|
||||
6. Log standard deviations are parameterized, not standard deviations, and state-independence
|
||||
is an implementation choice that differs across the algorithms in this book.
|
||||
|
||||
## Connects To
|
||||
- **Ch 8**: the taxonomy built on what each algorithm chooses to learn.
|
||||
- **Ch 9**: where the advantage function becomes the policy gradient weight.
|
||||
- **Ch 17**: the continuous-action argmax problem, straight out of `a*(s) = argmax_a Q*(s,a)`.
|
||||
- **Ch 12**: Exercise 1.1 asks you to implement the diagonal Gaussian log-likelihood above.
|
||||
|
|
@ -0,0 +1,123 @@
|
|||
# Chapter 8: Part 2 — Kinds of RL Algorithms
|
||||
|
||||
## Core Idea
|
||||
Two branching questions organize modern RL: does the agent have or learn a **model** of the
|
||||
environment, and **what does it learn** — a policy, a Q-function, a value function, or a
|
||||
model. Every trade-off in algorithm design descends from those two choices.
|
||||
|
||||
## Frameworks Introduced
|
||||
- **Branch 1: model-free vs model-based.** A model is a function predicting state transitions
|
||||
and rewards.
|
||||
- **Upside of a model: it allows the agent to plan** — think ahead, see what would happen
|
||||
across options, explicitly decide, then distil the planning results into a learned policy.
|
||||
AlphaZero is the famous case. When it works, it substantially improves sample efficiency.
|
||||
- **Downside: a ground-truth model is usually not available**, so it must be learned from
|
||||
experience, and **bias in a learned model gets exploited by the agent** — producing an
|
||||
agent that performs well against its own model and sub-optimally, or terribly, in the
|
||||
real environment. Model-learning is fundamentally hard; large amounts of time and compute
|
||||
can fail to pay off.
|
||||
- When to use model-based: you have a reliable model, or sample efficiency dominates and
|
||||
you can afford the model-bias risk.
|
||||
|
||||
- **Branch 2: what to learn in model-free RL — two families.**
|
||||
- **Policy Optimization.** Represent the policy explicitly as `pi_theta(a|s)` and optimize
|
||||
theta either directly by gradient ascent on `J(pi_theta)` or indirectly by maximizing
|
||||
local approximations of it. Almost always **on-policy**. Usually also learns an
|
||||
approximator `V_phi(s)` used in figuring out the policy update. Examples: A2C/A3C
|
||||
(direct), PPO (indirect, via a surrogate objective giving a conservative estimate of how
|
||||
much `J` will change).
|
||||
- **Q-Learning.** Learn an approximator `Q_theta(s,a)` to `Q*(s,a)`, usually with an
|
||||
objective based on the Bellman equation. Almost always **off-policy**. The policy comes
|
||||
from the connection `a(s) = argmax_a Q_theta(s,a)`. Examples: DQN, C51 (which learns a
|
||||
distribution over return whose expectation is `Q*`).
|
||||
|
||||
- **The policy-optimization vs Q-learning trade-off** (the single most quoted judgment in
|
||||
this book):
|
||||
- Policy optimization is **principled — you directly optimize for the thing you want** —
|
||||
which tends to make it stable and reliable.
|
||||
- Q-learning only *indirectly* optimizes agent performance, by training `Q_theta` to satisfy
|
||||
a self-consistency equation. **There are many failure modes for this kind of learning, so
|
||||
it tends to be less stable.**
|
||||
- But when Q-learning works it is **substantially more sample efficient**, because it can
|
||||
reuse data far more effectively.
|
||||
- How to use: pick the failure you can detect and afford. Instability you can see in a
|
||||
learning curve; sample inefficiency you can see in a compute bill.
|
||||
|
||||
- **The interpolation principle**: policy optimization and Q-learning are not incompatible,
|
||||
and under some circumstances turn out to be *equivalent* (Schulman et al 2017). A range of
|
||||
algorithms lives between them and trades off deliberately: **DDPG** (learns a deterministic
|
||||
policy and a Q-function that improve each other) and **SAC** (a variant using stochastic
|
||||
policies, entropy regularization and other tricks to stabilize learning; scores higher than
|
||||
DDPG on standard benchmarks).
|
||||
|
||||
- **Four ways to use a model** (there is no small set of clean clusters here; the model may
|
||||
be given or learned in each):
|
||||
1. **Pure planning** — never represent the policy at all. Model-predictive control (MPC):
|
||||
each time the agent observes, compute a plan optimal with respect to the model over a
|
||||
fixed window, execute only the first action, discard the rest, re-plan next step. Future
|
||||
rewards past the horizon can enter through a learned value function. Example: MBMF.
|
||||
2. **Expert iteration** — keep an explicit policy `pi_theta`, use a planning algorithm
|
||||
(e.g. Monte Carlo Tree Search) inside the model with candidate actions sampled from the
|
||||
current policy. The planner's output is an "expert" relative to the policy; update the
|
||||
policy toward it. Examples: ExIt, AlphaZero.
|
||||
3. **Data augmentation for model-free methods** — train a policy or Q-function with a
|
||||
model-free algorithm, but augment real experience with fictitious experience (MBVE), or
|
||||
train on purely fictitious experience ("training in the dream", World Models).
|
||||
4. **Embedding planning loops into policies** — make the planning procedure a subroutine of
|
||||
the policy so complete plans become side information, and train the policy output with
|
||||
any standard model-free algorithm. **The key advantage: model bias becomes less of a
|
||||
problem, because where the model is bad for planning the policy can learn to ignore it.**
|
||||
Example: I2A.
|
||||
|
||||
## Key Concepts
|
||||
- **Model of the environment**: a function predicting state transitions and rewards.
|
||||
- **Surrogate objective**: a local approximation of `J(pi_theta)` that is safe to maximize;
|
||||
the mechanism behind TRPO and PPO.
|
||||
- **The deadly triad**: function approximation + bootstrapping + off-policy data, which
|
||||
together cause instability in value-learning algorithms (Sutton and Barto ch. 11.3). This
|
||||
is the concrete content of "Q-learning has many failure modes."
|
||||
- **Modularity caveat**: the taxonomy is a tree and the real space is not. Advanced areas —
|
||||
exploration, transfer learning, meta learning — are omitted from it entirely.
|
||||
|
||||
## Reference Tables
|
||||
|
||||
| Choice | Buys you | Costs you |
|
||||
|--------|----------|-----------|
|
||||
| Model-based | Planning; large sample-efficiency gains when the model is good | Model must usually be learned; model bias gets exploited |
|
||||
| Model-free | Easier to implement and tune; more developed and tested | Forfeits the sample-efficiency gains a model could give |
|
||||
| Policy optimization | Stability, reliability, directly optimizes performance | Sample inefficiency (on-policy data only) |
|
||||
| Q-learning | Substantial sample efficiency through data reuse | Instability; no guarantee good Bellman fit means good policy |
|
||||
|
||||
## Mental Models
|
||||
- **Place any new algorithm with two questions**: model or no model, and what does it learn.
|
||||
That is what the taxonomy is for; it does not need to be exhaustive to do that job.
|
||||
- **Model bias is an adversarial problem, not a noise problem.** The agent is actively
|
||||
optimizing against your model's errors. That is why "the model is only 95% accurate" does
|
||||
not translate into "the policy is 95% as good."
|
||||
- **"Directly optimize the thing you want" is a design principle you can apply outside RL**:
|
||||
the further your training objective sits from your evaluation metric, the more failure modes
|
||||
you inherit.
|
||||
|
||||
## Anti-patterns
|
||||
- **Reading the taxonomy as exhaustive or as a strict tree.** The book opens with that
|
||||
disclaimer: algorithm modularity is not well represented by a tree.
|
||||
- **Assuming model-based is strictly better because it plans.** As of the guide's writing
|
||||
(September 2018), model-free methods were more popular and more extensively developed and
|
||||
tested, precisely because model-learning is hard.
|
||||
- **Treating "it fits the Bellman equation well" as evidence the policy is good.** No such
|
||||
guarantee exists — this is the defining weakness of the whole Q-learning family.
|
||||
|
||||
## Key Takeaways
|
||||
1. Two branching questions — model or not, and what to learn — generate the whole landscape.
|
||||
2. Policy optimization: principled, stable, sample-hungry. Q-learning: sample-efficient,
|
||||
indirect, unstable.
|
||||
3. The two families are not exclusive; DDPG and SAC live between them on purpose.
|
||||
4. Model bias is exploited by the agent, which is the central risk of model-based methods.
|
||||
5. Embedding a planner inside a policy is the model-based approach that most directly
|
||||
defuses model bias, because the policy can learn when to ignore the model.
|
||||
|
||||
## Connects To
|
||||
- **Ch 3**: the six implemented algorithms, which are all model-free.
|
||||
- **Ch 9**: the mathematics of the policy-optimization family.
|
||||
- **Ch 17 and Ch 19**: DDPG and SAC as the concrete interpolations named here.
|
||||
- **Ch 11**: the key-papers list, whose top-level sections mirror this taxonomy.
|
||||
|
|
@ -0,0 +1,161 @@
|
|||
# Chapter 9: Part 3 — Intro to Policy Optimization
|
||||
|
||||
## Core Idea
|
||||
Three results build the policy gradient you actually implement: the simplest analytical
|
||||
expression for `grad J`, a rule that lets you **drop useless terms** (reward-to-go), and a
|
||||
rule that lets you **add useful terms** (baselines) — ending at the advantage-weighted form
|
||||
`grad J = E[ sum_t grad log pi_theta(a_t|s_t) * A^pi(s_t,a_t) ]`.
|
||||
|
||||
## Frameworks Introduced
|
||||
- **The general policy gradient form**: every variant is
|
||||
`grad_theta J(pi_theta) = E_{tau~pi_theta}[ sum_{t=0}^{T} grad_theta log pi_theta(a_t|s_t) * Phi_t ]`
|
||||
and the whole subject is *which* `Phi_t` you choose. Five valid choices, all with the same
|
||||
expectation and different variance:
|
||||
1. `Phi_t = R(tau)` — the full trajectory return (the simplest form)
|
||||
2. `Phi_t = sum_{t'=t}^{T} R(s_t', a_t', s_{t'+1})` — the **reward-to-go**
|
||||
3. `Phi_t = reward-to-go - b(s_t)` — reward-to-go with a **baseline**
|
||||
4. `Phi_t = Q^{pi_theta}(s_t, a_t)`
|
||||
5. `Phi_t = A^{pi_theta}(s_t, a_t)` — the **advantage**, the one Spinning Up's VPG uses
|
||||
- When to use: choice 5 in practice; the list is the map for reading any policy gradient paper.
|
||||
|
||||
- **The derivation, in five facts.** Worth memorizing because every extension reuses them:
|
||||
1. `P(tau|theta) = rho_0(s_0) * prod_t P(s_{t+1}|s_t,a_t) * pi_theta(a_t|s_t)`
|
||||
2. **Log-derivative trick**: `grad_theta P(tau|theta) = P(tau|theta) * grad_theta log P(tau|theta)`
|
||||
3. `log P(tau|theta) = log rho_0(s_0) + sum_t [ log P(s_{t+1}|s_t,a_t) + log pi_theta(a_t|s_t) ]`
|
||||
4. **Gradients of environment functions are zero** — the environment has no dependence on theta,
|
||||
so `rho_0`, `P` and `R` all vanish under `grad_theta`.
|
||||
5. Therefore `grad_theta log P(tau|theta) = sum_t grad_theta log pi_theta(a_t|s_t)`.
|
||||
- Chained: expand the expectation, bring the gradient inside the integral, apply the
|
||||
log-derivative trick, return to expectation form, substitute fact 5.
|
||||
|
||||
- **The EGLP lemma (Expected Grad-Log-Prob)**: for any parameterized distribution `P_theta`,
|
||||
`E_{x~P_theta}[ grad_theta log P_theta(x) ] = 0`.
|
||||
- Proof in three lines: all distributions are normalized (`integral P_theta(x) = 1`); take
|
||||
the gradient of both sides (`= grad 1 = 0`); apply the log-derivative trick.
|
||||
- When to use: it is the engine behind both the drop rule and the add rule. The author notes
|
||||
it has no standard name in the literature but comes up often enough to deserve one.
|
||||
|
||||
- **"Don't let the past distract you" (the drop rule)**: agents should only reinforce actions
|
||||
on the basis of their **consequences**. Rewards obtained *before* an action have no bearing
|
||||
on how good that action was. Formally, all terms with `t' < t` are zero in expectation.
|
||||
- Why it is better, precisely: those dropped terms had **zero mean but nonzero variance** —
|
||||
they added pure noise to the sample estimate. Removing them reduces the number of sample
|
||||
trajectories needed.
|
||||
|
||||
- **Baselines (the add rule)**: an immediate consequence of EGLP is that for any function `b`
|
||||
depending only on state, `E_{a_t~pi}[ grad log pi(a_t|s_t) * b(s_t) ] = 0`. So you may add
|
||||
or subtract any such term without changing the gradient in expectation.
|
||||
- **The most common baseline is the on-policy value function `V^pi(s_t)`**, which empirically
|
||||
reduces variance and gives faster, more stable learning.
|
||||
- The conceptual appeal: it encodes the intuition that **if an agent gets what it expected,
|
||||
it should "feel" neutral about it.**
|
||||
- In practice `V^pi` cannot be computed exactly, so it is approximated by a network `V_phi`
|
||||
updated concurrently with the policy (so it always approximates the *most recent* policy).
|
||||
The simplest learning rule, used by VPG, TRPO, PPO and A2C, is mean-squared error:
|
||||
`phi_k = argmin_phi E_{s_t, Rhat_t ~ pi_k}[ (V_phi(s_t) - Rhat_t)^2 ]`, via one or more
|
||||
gradient steps starting from `phi_{k-1}`.
|
||||
|
||||
## Key Concepts
|
||||
- **Policy gradient**: `grad_theta J(pi_theta)`. Algorithms that optimize this way are
|
||||
policy gradient algorithms — VPG and TRPO are; **PPO is often called one though this is
|
||||
slightly inaccurate.**
|
||||
- **Reward-to-go**: `Rhat_t = sum_{t'=t}^{T} R(s_t', a_t', s_{t'+1})`.
|
||||
- **Baseline**: any state-only function subtracted from the weight.
|
||||
- **Sample estimate**: `ghat = (1/|D|) * sum_{tau in D} sum_t grad log pi_theta(a_t|s_t) * Phi_t`.
|
||||
- **GAE (Generalized Advantage Estimation)**: the widely-used method for approximating the
|
||||
advantage function; Spinning Up's VPG, TRPO and PPO all use it. The book strongly advises
|
||||
studying the paper.
|
||||
- **Epoch** (in this context): one experience-collection phase plus one policy gradient update.
|
||||
|
||||
## Code Examples
|
||||
The whole simple algorithm is 128 lines (`spinup/examples/pytorch/pg_math/1_simple_pg.py`).
|
||||
The three pieces that matter:
|
||||
|
||||
```python
|
||||
# 1. Policy network
|
||||
logits_net = mlp(sizes=[obs_dim] + hidden_sizes + [n_acts])
|
||||
|
||||
def get_policy(obs):
|
||||
return Categorical(logits=logits_net(obs))
|
||||
|
||||
def get_action(obs):
|
||||
return get_policy(obs).sample().item()
|
||||
|
||||
# 2. The "loss" whose gradient is the policy gradient
|
||||
def compute_loss(obs, act, weights):
|
||||
logp = get_policy(obs).log_prob(act)
|
||||
return -(logp * weights).mean()
|
||||
|
||||
# 3. One gradient step
|
||||
optimizer.zero_grad()
|
||||
batch_loss = compute_loss(obs=..., act=..., weights=...)
|
||||
batch_loss.backward()
|
||||
optimizer.step()
|
||||
```
|
||||
|
||||
Upgrading to reward-to-go changes one function and two lines:
|
||||
|
||||
```python
|
||||
def reward_to_go(rews):
|
||||
n = len(rews)
|
||||
rtgs = np.zeros_like(rews)
|
||||
for i in reversed(range(n)):
|
||||
rtgs[i] = rews[i] + (rtgs[i+1] if i+1 < n else 0)
|
||||
return rtgs
|
||||
|
||||
# was: batch_weights += [ep_ret] * ep_len
|
||||
batch_weights += list(reward_to_go(ep_rews))
|
||||
```
|
||||
|
||||
For a diagonal Gaussian policy, `log_prob(act)` returns per-component log probabilities of
|
||||
shape `(batch, act_dim)` when RL needs shape `(batch,)`. Sum them:
|
||||
|
||||
```python
|
||||
logp = get_policy(obs).log_prob(act).sum(axis=-1)
|
||||
```
|
||||
|
||||
## Worked Example
|
||||
**The policy gradient "loss" is not a loss function.** This is the single most consequential
|
||||
warning in the chapter, and it differs from supervised learning in two ways:
|
||||
|
||||
1. **The data distribution depends on the parameters.** A supervised loss is defined on a
|
||||
fixed data distribution independent of the parameters being optimized. Here the data must
|
||||
be sampled from the most recent policy.
|
||||
2. **It does not measure performance.** We care about `J(pi_theta)`, and this "loss" does not
|
||||
approximate it — not even in expectation. It is useful only because, *evaluated at the
|
||||
current parameters with data generated by those parameters*, it has the negative gradient
|
||||
of performance.
|
||||
|
||||
After the first gradient step there is no connection to performance at all. You can send
|
||||
this loss to negative infinity while policy performance craters — and it usually will.
|
||||
Researchers sometimes call this the policy "overfitting" to a batch; the phrase is
|
||||
descriptive but should not be taken literally, since it does not refer to generalization error.
|
||||
|
||||
**The practical rule: in policy gradients, only average return means anything. The loss
|
||||
function means nothing.** The ML habit of reading "loss went down, all is well" is wrong here.
|
||||
|
||||
## Anti-patterns
|
||||
- **Using the loss curve as a training health signal.** See above. Watch `AverageEpRet`.
|
||||
- **Weighting every action by the full-trajectory return** when reward-to-go is a two-line
|
||||
change that strictly reduces variance.
|
||||
- **Forgetting to sum log-probabilities across action dimensions** for Gaussian policies —
|
||||
a shape bug that will not raise, in the same family as the ch12 DDPG bug.
|
||||
- **Treating a baseline as a bias**: baselines are provably zero-mean under EGLP. They change
|
||||
variance, never the expected gradient.
|
||||
|
||||
## Key Takeaways
|
||||
1. One template, five valid weights `Phi_t`; the advantage form is the destination.
|
||||
2. The whole derivation rests on the log-derivative trick plus the fact that environment
|
||||
gradients vanish.
|
||||
3. EGLP is a single lemma that licenses both dropping past rewards and adding baselines.
|
||||
4. Reward-to-go removes terms with zero mean and nonzero variance — pure noise reduction.
|
||||
5. The value baseline is learned by mean-squared regression onto reward-to-go, concurrently
|
||||
with the policy.
|
||||
6. The policy gradient loss is not a performance measure and must never be read as one.
|
||||
|
||||
## Connects To
|
||||
- **Ch 7**: the advantage function and value functions this chapter puts to work.
|
||||
- **Ch 14**: VPG, the direct implementation of the final advantage-weighted form.
|
||||
- **Ch 15 and Ch 16**: TRPO and PPO, which replace the plain gradient step with a
|
||||
trust-region-constrained one.
|
||||
- **Ch 12**: Exercise 2.1 measures exactly what happens when `V_phi` is not fit properly.
|
||||
|
|
@ -0,0 +1,133 @@
|
|||
# Chapter 10: Spinning Up as a Deep RL Researcher
|
||||
|
||||
## Core Idea
|
||||
Joshua Achiam's curriculum for becoming a deep RL researcher, in four stages: build the
|
||||
right background, learn by writing your own implementations, develop a research project
|
||||
through one of three idea frames, and hold yourself to experimental standards strict enough
|
||||
that a positive result means something.
|
||||
|
||||
## Frameworks Introduced
|
||||
- **The Right Background** (four items, all prerequisites, none optional):
|
||||
- **Math**: from probability and statistics — random variables, Bayes' theorem, chain rule
|
||||
of probability, expected values, standard deviations, importance sampling. From
|
||||
multivariate calculus — gradients, and optionally Taylor series expansions.
|
||||
- **General deep learning**: standard architectures (MLP, vanilla RNN, LSTM, GRU, conv
|
||||
layers, resnets, attention), regularizers (weight decay, dropout), normalization (batch,
|
||||
layer, weight norm), optimizers (SGD, momentum SGD, Adam), and the reparameterization trick.
|
||||
- **One deep learning library**, well enough to confidently implement a simple supervised
|
||||
learning program.
|
||||
- **RL concepts and terminology**: states, actions, trajectories, policies, rewards, value
|
||||
functions, action-value functions. Optionally, monotonic improvement theory (the basis
|
||||
for advanced policy gradient algorithms) and classical RL algorithms.
|
||||
|
||||
- **Learn by Doing** — the core of the essay, in eight rules:
|
||||
1. **Write your own implementations**, aiming for the **shortest correct implementation** of
|
||||
each. By far the best way to build both understanding and performance intuitions.
|
||||
2. **Simplicity is critical.** Implement the simplest algorithms first and add complexity
|
||||
gradually. Starting with too many moving parts means weeks lost to debugging — a common
|
||||
failure mode for newcomers. If you are stuck in it, drop to a simpler algorithm and
|
||||
return later.
|
||||
3. **Which algorithms, in roughly this order**: VPG (REINFORCE), DQN, A2C (the synchronous
|
||||
A3C), PPO (clipped objective), DDPG. The simplest versions are a few hundred lines
|
||||
(ballpark 250-300); a no-frills VPG is about 80. Write single-threaded before parallel,
|
||||
but do parallelize at least one.
|
||||
4. **Focus on understanding, because broken RL code almost always fails silently** — the
|
||||
code runs fine and the agent simply never learns. Usually something is computed with the
|
||||
wrong equation, on the wrong distribution, or piped to the wrong place. Sometimes the
|
||||
only way to find it is to read the code critically, knowing exactly what it should do.
|
||||
5. **What to look for in papers**: scour ablation analyses (they tell you which parameters
|
||||
and subroutines actually matter, which is what you need to diagnose bugs) and
|
||||
supplementary material (network architectures, optimization hyperparameters — align your
|
||||
implementation to these to improve your odds of getting it working).
|
||||
6. **But do not overfit to paper details.** Papers often prescribe more tricks than are
|
||||
strictly necessary. The original DDPG paper suggests a complex architecture, an
|
||||
initialization scheme and batch normalization; none are strictly necessary and some of
|
||||
the best DDPG results use simpler networks. The original A3C uses asynchronous updates
|
||||
from actor-learners; synchronous updates work about as well.
|
||||
7. **Do not overfit to existing implementations either.** RL libraries make abstraction
|
||||
choices that are good for code reuse across algorithms but unnecessary if you are
|
||||
writing one algorithm for one use case.
|
||||
8. **Iterate fast in simple environments.** CartPole-v0, InvertedPendulum-v0, FrozenLake-v0,
|
||||
HalfCheetah-v2 with a short horizon (100 or 250 steps, not the full 1000).
|
||||
**Ideal debug-stage turnaround: under 5 minutes on your local machine.** These runs need
|
||||
no special hardware and run fine on CPUs. Do not attempt Atari or Humanoid before the
|
||||
simplest toy task works.
|
||||
|
||||
- **Three idea-generation frames** — the frame you pick shapes the project's scope and risk:
|
||||
|
||||
| Frame | What it is | Scope | Main risk |
|
||||
|-------|-----------|-------|-----------|
|
||||
| **1. Improving on an existing approach** | Incrementalism: performance gains in an established setting by tweaking an existing algorithm. Reimplementing prior work exposes where it is brittle. Most accessible to novices, still worthwhile at any level | Narrow, wraps up in a few months | Your tweaks may simply fail to improve it, and then the project is over with no signal on what to do next |
|
||||
| **2. Focusing on unsolved benchmarks** | Succeed at a task nobody has solved. You may try a wide variety of methods, prior and invented | Broad, several months to a year-plus | The benchmark may be unsolvable without a substantial breakthrough; easy to spend a long time with no progress. Even failure usually yields insights that seed the next project |
|
||||
| **3. Creating a new problem setting** | An entirely different conceptual problem nobody has studied; you will have to design the benchmark too | Open-ended | Enormous challenge — but great benchmarks move the whole field forward. These problems come up when they come up; hard to go looking for them |
|
||||
|
||||
- **Doing Rigorous Research in RL** — four standards, each closing a specific way a result
|
||||
can look real and not be:
|
||||
1. **Set up fair comparisons.** If you implement your baseline from scratch, spend **as much
|
||||
time tuning the baseline as tuning your own algorithm.** Hold all else equal even when
|
||||
the methods differ substantially — e.g. keep parameter counts approximately equal when
|
||||
investigating architecture variants. **Under no circumstances handicap the baseline.**
|
||||
RL baselines are strong, and consistent wins over them are hard.
|
||||
2. **Remove stochasticity as a confounder.** Run everything for many random seeds — **at
|
||||
least 3, and 10 or more to be thorough.** Deep RL is fairly brittle with respect to seed;
|
||||
two different groups of seeds can produce learning curves so different they look like
|
||||
they come from different distributions.
|
||||
3. **Run high-integrity experiments.** Do not report the best or most interesting runs.
|
||||
Launch **new, final experiments** for every method being compared and **precommit to
|
||||
reporting whatever comes out.** This enforces a weak form of preregistration: the tuning
|
||||
stage produces your hypotheses, the final runs produce your conclusions.
|
||||
4. **Check each claim separately — run an ablation analysis.** Any proposed method has
|
||||
several key design decisions, and the claim "these collectively help" is really a bundle
|
||||
of separate claims. Systematically swap or remove each one to attribute credit correctly.
|
||||
This lets you state each claim with a measure of confidence and strengthens the whole work.
|
||||
|
||||
## Key Concepts
|
||||
- **Silent failure**: the defining property of broken RL code. It runs; the agent just never learns.
|
||||
- **Measure everything**: instrument heavily. The author's own list — mean/std/min/max of
|
||||
cumulative rewards, episode lengths and value function estimates, plus the objective losses
|
||||
and any exploration parameters (mean policy entropy for stochastic policy optimization,
|
||||
current epsilon for epsilon-greedy). **Also watch videos of your agent** periodically; it
|
||||
gives insights nothing else does. You cannot tell it is broken if you cannot see it breaking.
|
||||
- **Assume there is a bug.** Spend a lot of effort searching for bugs before tweaking
|
||||
hyperparameters. Bad hyperparameters can significantly degrade performance, but if yours
|
||||
are similar to those in papers and standard implementations, they are probably not the issue.
|
||||
- **Test in more than one environment**: sometimes code works in one environment despite a
|
||||
breaking bug, so re-verify once results look promising.
|
||||
- **Avoid reinventing the wheel**: before investing, check thoroughly that the idea has not
|
||||
been done. But do not let the risk push you into planting flags with not-quite-finished
|
||||
research or over-claiming partial work. Complete, thorough investigations are what counts.
|
||||
- **Scale when things work**: after the simplest environments pass, move to harder ones —
|
||||
experiments now take hours to a couple of days, and specialized hardware (a beefy GPU, a
|
||||
32-core machine) or cloud resources start to be worth it.
|
||||
|
||||
## Anti-patterns
|
||||
- **Tuning hyperparameters to fix a bug.** It is usually a bug.
|
||||
- **Debugging in a hard environment.** If turnaround exceeds a few minutes at the debug stage,
|
||||
you are working in the wrong environment.
|
||||
- **Starting with the complex algorithm.** The predicted outcome is weeks lost.
|
||||
- **Reporting your best run.** That is the failure the precommitment rule exists to stop.
|
||||
- **Under-tuning the baseline** — the most common way to manufacture a positive result without
|
||||
intending to.
|
||||
- **Fewer than 3 seeds.** With deep RL's seed sensitivity, a single-seed result carries
|
||||
almost no information.
|
||||
- **Bundling claims.** Without ablations you cannot say which of your design decisions did
|
||||
the work — including, possibly, none of them.
|
||||
|
||||
## Key Takeaways
|
||||
1. Implement from scratch, simplest first, shortest correct version, single-threaded before parallel.
|
||||
2. Broken RL code fails silently — so instrumentation and critical reading, not error messages,
|
||||
are your debugging tools.
|
||||
3. Debug loops under five minutes in toy environments; scale only after correctness.
|
||||
4. Read papers for ablations and supplementary material, but do not adopt every trick.
|
||||
5. Pick an idea frame deliberately: incremental (narrow, fast, may dead-end), unsolved
|
||||
benchmark (broad, slow, insight-rich even in failure), or new problem setting (rare, huge).
|
||||
6. Rigor is four things: a fully-tuned baseline, many seeds, precommitted final runs, and
|
||||
per-claim ablations.
|
||||
7. These habits are worth keeping past the learning stage; they accelerate research.
|
||||
|
||||
## Connects To
|
||||
- **Ch 1**: the code design philosophy that makes the reference implementations readable.
|
||||
- **Ch 4**: the `--seed 0 10 20` flag that operationalizes the multi-seed rule.
|
||||
- **Ch 11**: the key papers list, the recommended starting point for literature exploration.
|
||||
- **Ch 12**: the exercises — Problem Set 2 is entirely about silent failure modes.
|
||||
- **Ch 13**: the benchmarks, run at 10 seeds — the standard this chapter asks for.
|
||||
|
|
@ -0,0 +1,70 @@
|
|||
# Chapter 11: Key Papers in Deep RL
|
||||
|
||||
## Core Idea
|
||||
A curated, explicitly non-comprehensive reading list of roughly 100 papers organized into
|
||||
13 topic areas — designed as a starting point for someone looking to do research, and as
|
||||
the map you use to pick a research topic in ch10's "explore the literature" step.
|
||||
|
||||
## Frameworks Introduced
|
||||
- **Use the list as a topic map, not a queue.** The intended workflow (from ch10): scan the
|
||||
sections to become aware of what topics exist, find a paper on one that inspires you, read
|
||||
it thoroughly, then use its related-work section and citations to do a deep dive. You will
|
||||
start to see where the unsolved problems are.
|
||||
- **Each entry is annotated with what it contributes** — either **Algorithm: X** (this paper
|
||||
introduced X) or **Contribution: ...** (this paper established a result, a critique or a
|
||||
codebase). Scan the annotations, not the titles.
|
||||
|
||||
## Reference Tables
|
||||
|
||||
The 13 sections, with the subsections that reveal the field's internal structure:
|
||||
|
||||
| # | Section | Subsections / notable entries |
|
||||
|---|---------|------------------------------|
|
||||
| 1 | **Model-Free RL** | a. Deep Q-Learning (DQN, Deep Recurrent Q-Learning, Dueling DQN, Double DQN, Prioritized Experience Replay, Rainbow) · b. Policy Gradients (A3C, TRPO, GAE, PPO, ACKTR, ACER, SAC) · c. Deterministic Policy Gradients (DPG, DDPG, TD3) · d. Distributional RL (C51, QR-DQN, IQN, Dopamine) · e. Policy Gradients with Action-Dependent Baselines (Q-Prop, Stein Control Variates, and Tucker et al's critique) · f. Path-Consistency Learning (PCL, Trust-PCL) · g. Other Ways of Combining Policy-Learning and Q-Learning (PGQL, Reactor, IPG, the policy-gradient/soft-Q-learning equivalence) · h. Evolutionary Algorithms (ES) |
|
||||
| 2 | **Exploration** | a. Intrinsic Motivation (VIME, count-based pseudocounts, hash-based counts, EX2, ICM, RND) · b. Unsupervised RL (VIC, DIAYN, VALOR) |
|
||||
| 3 | **Transfer and Multitask RL** | Progressive Networks, UVFA, UNREAL, PathNet, MATL, HER |
|
||||
| 4 | **Hierarchy** | STRAW, Feudal Networks, HIRO |
|
||||
| 5 | **Memory** | MFEC, NEC, Neural Map, MERLIN, RMC |
|
||||
| 6 | **Model-Based RL** | a. Model is Learned (I2A, MBMF, MVE) · b. Model is Given (AlphaZero and relatives) |
|
||||
| 7 | **Meta-RL** | |
|
||||
| 8 | **Scaling RL** | |
|
||||
| 9 | **RL in the Real World** | |
|
||||
| 10 | **Safety** | |
|
||||
| 11 | **Imitation Learning and Inverse Reinforcement Learning** | |
|
||||
| 12 | **Reproducibility, Analysis, and Critique** | |
|
||||
| 13 | **Bonus: Classic Papers in RL Theory or Review** | |
|
||||
|
||||
## Mental Models
|
||||
- **Section 1's shape mirrors ch8's taxonomy**: Q-learning, policy gradients, and the
|
||||
deterministic and interpolating methods between them. If you understood ch8, section 1
|
||||
is already organized in your head.
|
||||
- **Section 12 is the one people skip and shouldn't.** "Reproducibility, Analysis, and
|
||||
Critique" is where ch10's rigor standards come from, and where you learn that published
|
||||
gains sometimes do not survive re-examination — the Tucker et al entry in section 1e is a
|
||||
worked example: it critiques and re-evaluates claims from earlier papers (including Q-Prop
|
||||
and Stein control variates) and finds important methodological errors in them.
|
||||
- **A reading list is a topic-awareness tool.** The list explicitly does not claim
|
||||
completeness; its job is to prevent you from picking a research problem while unaware that
|
||||
a whole subfield already works on it.
|
||||
|
||||
## Anti-patterns
|
||||
- **Reading it front to back.** It is far from comprehensive and is not a curriculum; the
|
||||
intended use is targeted depth after topic selection.
|
||||
- **Treating an entry as endorsement of the result.** Several entries exist precisely because
|
||||
they critique other entries.
|
||||
- **Skipping the related-work walk.** The list is the entry point; the citations are the
|
||||
actual literature review.
|
||||
|
||||
## Key Takeaways
|
||||
1. 13 topic sections; section 1 (Model-Free RL) alone has eight subsections and mirrors ch8.
|
||||
2. Every entry is annotated with the algorithm it introduced or the contribution it made.
|
||||
3. The list is a topic map for choosing a research direction, not a reading queue.
|
||||
4. Sections 7-13 (meta-RL, scaling, real world, safety, imitation/IRL, critique, classics)
|
||||
are where the topics ch10 suggests for project ideas actually live.
|
||||
|
||||
## Connects To
|
||||
- **Ch 8**: the taxonomy that section 1's structure reproduces.
|
||||
- **Ch 10**: "start by exploring the literature to become aware of topics in the field" —
|
||||
this chapter is that step's tool.
|
||||
- **Ch 14-19**: every implemented algorithm's own "Why These Papers?" section is a
|
||||
three-paper version of this list, scoped to one algorithm.
|
||||
|
|
@ -0,0 +1,105 @@
|
|||
# Chapter 12: Exercises
|
||||
|
||||
## Core Idea
|
||||
Two problem sets with opposite purposes: Problem Set 1 makes you write the mathematical
|
||||
core of three algorithms, and Problem Set 2 makes you **watch RL code fail silently** and
|
||||
find out why — because that is the skill the essay says decides whether you can do this work.
|
||||
|
||||
## Frameworks Introduced
|
||||
- **Problem Set 1: Basics of Implementation** — you are given everything except the math.
|
||||
- **1.1 Gaussian Log-Likelihood.** Write a function taking means, log stds and samples,
|
||||
returning the log likelihoods. Auto-checked against a known-good implementation on a
|
||||
batch of random inputs. The formula is the one in ch7.
|
||||
- **1.2 Policy for PPO.** Implement an MLP diagonal Gaussian policy. Auto-checked by
|
||||
running 20 epochs on InvertedPendulum-v2 (3-5 minutes). **Bar for success: average score
|
||||
over 500 in the last 5 epochs, or 1000 (the maximum) in the last 5.**
|
||||
- **1.3 Computation Graph for TD3.** You are given the entire TD3 algorithm *except* the
|
||||
loss functions and the intermediate calculations for them. Find "YOUR CODE HERE".
|
||||
No automatic checking. Evaluated on HalfCheetah-v2, InvertedPendulum-v2 and one other
|
||||
MuJoCo environment of your choosing, with smaller networks ([128,128]), max episode
|
||||
length 150, 10 epochs, roughly 10 minutes. **Anecdotal targets within 10 epochs:
|
||||
HalfCheetah over 300, InvertedPendulum maxing out at 150.** `--use_soln` runs Spinning
|
||||
Up's TD3 instead of yours.
|
||||
|
||||
- **Problem Set 2: Algorithm Failure Modes** — the point is the failure, not the fix.
|
||||
- **2.1 Value Function Fitting in TRPO.** Compare `train_v_iters=80` against
|
||||
`train_v_iters=0` on Hopper-v2, three seeds each, 250 epochs, 4000 steps per epoch.
|
||||
**Result: the difference is substantial. With a trained value function the agent makes
|
||||
quick progress; with an untrained one it gets stuck early on.** Few factors affect policy
|
||||
gradient performance more drastically than the quality of the value function used for
|
||||
advantage estimation.
|
||||
- **2.2 Silent Bug in DDPG.** Run DDPG with and without a planted bug, three seeds each,
|
||||
six runs, ~10 minutes each; plot and compare. Then, **without looking at DDPG's `core.py`**,
|
||||
work out what the bug is.
|
||||
|
||||
## Worked Example
|
||||
**The 2.2 bug, and why it is the most instructive page in the book.**
|
||||
|
||||
The correct and bugged actor-critic differ in exactly one thing: whether the Q-function
|
||||
output is squeezed.
|
||||
|
||||
```python
|
||||
# Correct
|
||||
def forward(self, obs, act):
|
||||
q = self.q(torch.cat([obs, act], dim=-1))
|
||||
return torch.squeeze(q, -1) # Critical to ensure q has right shape.
|
||||
|
||||
# Bugged
|
||||
def forward(self, obs, act):
|
||||
return self.q(torch.cat([obs, act], dim=-1)) # shape [batch, 1], not [batch]
|
||||
```
|
||||
|
||||
The TF1 version is the same defect: the correct code squeezes to shape `[batch size]`, the
|
||||
bugged code leaves shape `[batch size, 1]`.
|
||||
|
||||
Why that one missing squeeze destroys learning — look at the DDPG graph:
|
||||
|
||||
```python
|
||||
backup = tf.stop_gradient(r_ph + gamma*(1-d_ph)*q_pi_targ)
|
||||
pi_loss = -tf.reduce_mean(q_pi)
|
||||
q_loss = tf.reduce_mean((q - backup)**2)
|
||||
```
|
||||
|
||||
`r_ph` and `d_ph` have shape `[batch size]`. The backup line was written assuming it adds
|
||||
tensors of the same shape. But it will also happily add tensors of *different* shapes as long
|
||||
as they are broadcast-compatible — and `[batch size]` and `[batch size, 1]` are compatible,
|
||||
with results that are not what you expect:
|
||||
|
||||
```
|
||||
x has shape [5], y has shape [5,1]
|
||||
x * y -> shape [5,5]
|
||||
x + y -> shape [5,5]
|
||||
```
|
||||
|
||||
Adding or multiplying a shape-`[5]` tensor by a shape-`[5,1]` tensor returns a shape-`[5,5]`
|
||||
tensor. So when the Q-functions are not squeezed, `q_pi_targ` has shape `[batch size, 1]`,
|
||||
the backup becomes a `[batch, batch]` matrix, and the whole Q-loss is meaningless. **Nothing
|
||||
raises. Nothing warns. The run completes. The agent just learns worse.**
|
||||
|
||||
This is ch10's "broken RL code almost always fails silently" reduced to a single missing
|
||||
`squeeze`. The **Bonus** question is worth sitting with: *are there any choices of
|
||||
hyperparameters which would have hidden the effects of the bug?*
|
||||
|
||||
## Anti-patterns
|
||||
- **Looking at `core.py` before attempting 2.2.** The exercise is diagnostic practice; the
|
||||
answer is worth less than the search.
|
||||
- **Broadcast-compatible shapes as an implicit contract.** The lesson generalizes far past
|
||||
DDPG: any elementwise op between a `[N]` and an `[N,1]` tensor is a silent bug waiting.
|
||||
Assert shapes.
|
||||
- **Skipping Problem Set 2 because it produces no code.** 2.1 produces no code at all and is
|
||||
still one of the most decision-relevant results in the book.
|
||||
|
||||
## Key Takeaways
|
||||
1. Problem Set 1 is math-in-code: Gaussian log-likelihood, a PPO policy, TD3's losses.
|
||||
2. 1.1 and 1.2 self-check; 1.3 does not, and is evaluated on visible learning progress.
|
||||
3. 2.1: a badly-fit value function does not degrade a policy gradient agent gracefully — it
|
||||
gets stuck early.
|
||||
4. 2.2: one missing `squeeze` silently corrupts the Bellman backup through broadcasting.
|
||||
5. The challenges past the problem sets are: reimplement algorithms from scratch (ch10), and
|
||||
attempt OpenAI's standing Requests for Research.
|
||||
|
||||
## Connects To
|
||||
- **Ch 7**: the diagonal Gaussian log-likelihood formula that Exercise 1.1 asks for.
|
||||
- **Ch 9**: why the value function baseline quality drives policy gradient performance (2.1).
|
||||
- **Ch 17 and Ch 18**: the DDPG graph and the TD3 losses the exercises operate on.
|
||||
- **Ch 10**: "broken RL code almost always fails silently" — Problem Set 2 is the proof.
|
||||
|
|
@ -0,0 +1,87 @@
|
|||
# Chapter 13: Benchmarks for Spinning Up Implementations
|
||||
|
||||
## Core Idea
|
||||
All six algorithms were benchmarked for 3M timesteps across five MuJoCo environments at
|
||||
10 seeds each — and the results carry an explicit, unusual disclosure about which
|
||||
implementations are research-grade and which are not.
|
||||
|
||||
## Frameworks Introduced
|
||||
- **The parity disclosure — the single most decision-relevant paragraph in the chapter:**
|
||||
- **DDPG, TD3 and SAC are roughly at parity with the best reported results** for those
|
||||
algorithms. **You can use them for research purposes.**
|
||||
- **VPG, TRPO and PPO are overall a bit weaker than the best reported results**, because
|
||||
the implementations omit standard tricks — specifically **observation normalization** and
|
||||
**normalized value regression targets**. **For research comparisons, use the TRPO or PPO
|
||||
implementations from OpenAI Baselines instead.**
|
||||
- When to use: before citing any Spinning Up number, or before using one of these as a
|
||||
baseline in a paper (which ch10 says you must tune as hard as your own method).
|
||||
|
||||
- **Report performance with its measurement definition attached.** The two families are not
|
||||
measured the same way, and saying "return" without saying which is a category error:
|
||||
- **On-policy**: the average trajectory return across the batch collected at each epoch —
|
||||
i.e. the exploring policy, measured continuously.
|
||||
- **Off-policy**: measured **once every 10,000 steps** by running the deterministic policy
|
||||
(or, for SAC, the **mean** policy) **without action noise** for **ten trajectories**, and
|
||||
reporting the average return over those test trajectories.
|
||||
|
||||
## Reference Tables
|
||||
|
||||
Benchmark setup:
|
||||
|
||||
| Parameter | On-policy (VPG, TRPO, PPO) | Off-policy (DDPG, TD3, SAC) |
|
||||
|-----------|---------------------------|----------------------------|
|
||||
| Network architecture | (64, 32), tanh units | (256, 256), relu units |
|
||||
| Batch | 4000 environment steps per batch update | Minibatches of 100 per gradient step |
|
||||
| Performance metric | Average batch trajectory return per epoch | Deterministic/mean policy, no noise, 10 trajectories every 10k steps |
|
||||
| Research-grade? | No — use Baselines for TRPO/PPO | Yes |
|
||||
|
||||
Common to all:
|
||||
|
||||
| Setting | Value |
|
||||
|---------|-------|
|
||||
| Environments | HalfCheetah-v3, Hopper-v3, Walker2d-v3, Swimmer-v3, Ant-v3 (MuJoCo Gym suite) |
|
||||
| Timesteps | 3M |
|
||||
| Random seeds | 10 per experiment |
|
||||
| Plot content | Solid line = mean over seeds; shaded = std dev over seeds |
|
||||
| Smoothing | Averaged over a window of 11 epochs |
|
||||
| Other hyperparameters | Left at Spinning Up defaults; see each algorithm page |
|
||||
|
||||
Both PyTorch and TF1 versions were benchmarked in every environment, plus dedicated
|
||||
head-to-head PyTorch-vs-TF1 pages for VPG, PPO, DDPG, TD3 and SAC. (TRPO has no PyTorch
|
||||
implementation, so it has no head-to-head page.)
|
||||
|
||||
## Mental Models
|
||||
- **Reporting mean and std dev over 10 seeds *as shaded bands* is the visual form of ch10's
|
||||
"remove stochasticity as a confounder."** A benchmark plot without a seed band is telling
|
||||
you less than it appears to.
|
||||
- **The architecture split is itself a finding**: on-policy at (64, 32)/tanh and off-policy
|
||||
at (256, 256)/relu is the convention these algorithm families settled into, not an
|
||||
arbitrary choice, and copying the wrong family's defaults is a real source of bad results.
|
||||
- **Honest self-assessment is a feature of the resource**, not a caveat. A benchmarks page
|
||||
that names which of its own implementations you should not use is doing the reader's
|
||||
ch10 rigor work for them.
|
||||
|
||||
## Anti-patterns
|
||||
- **Using Spinning Up's PPO or TRPO as a paper baseline.** The page says explicitly to use
|
||||
Baselines for those. Under-powered baselines are the ch10 failure of "handicapping the
|
||||
baseline," even when it is unintentional.
|
||||
- **Comparing an on-policy `AverageEpRet` against an off-policy `AverageTestEpRet`** as if
|
||||
they were the same quantity. Use the plotter's `Performance` alias (ch6), which resolves
|
||||
per family.
|
||||
- **Reading a single-seed curve as a result.** These plots are 10 seeds precisely because
|
||||
fewer is not informative.
|
||||
|
||||
## Key Takeaways
|
||||
1. 3M timesteps, five MuJoCo environments, 10 seeds, both backends.
|
||||
2. DDPG, TD3, SAC: research-usable. VPG, TRPO, PPO: educational only — use Baselines.
|
||||
3. The weakness is attributed to two named omissions: observation normalization and
|
||||
normalized value regression targets.
|
||||
4. On-policy and off-policy performance are different measurements, not different numbers
|
||||
of the same measurement.
|
||||
5. Architecture defaults differ by family: (64,32)/tanh on-policy, (256,256)/relu off-policy.
|
||||
|
||||
## Connects To
|
||||
- **Ch 1**: "reasonably good" — this chapter is the quantification of that phrase.
|
||||
- **Ch 6**: the `Performance` alias, which exists because of the metric split described here.
|
||||
- **Ch 10**: the seed and fair-baseline standards this benchmark actually meets.
|
||||
- **Ch 14-19**: per-algorithm hyperparameter defaults referenced as "left at default settings."
|
||||
|
|
@ -0,0 +1,93 @@
|
|||
# Chapter 14: Vanilla Policy Gradient (VPG)
|
||||
|
||||
## Core Idea
|
||||
Push up the probabilities of actions that lead to higher return and push down the
|
||||
probabilities of actions that lead to lower return, until you arrive at the optimal policy.
|
||||
That is the whole idea underlying policy gradients, and VPG is its most direct expression.
|
||||
|
||||
## Quick Facts
|
||||
- **On-policy.**
|
||||
- Works with **discrete or continuous** action spaces.
|
||||
- The Spinning Up implementation **supports MPI parallelization**.
|
||||
|
||||
## Key Equations
|
||||
The advantage-weighted policy gradient (the end point of ch9):
|
||||
|
||||
```
|
||||
grad_theta J(pi_theta) = E_{tau ~ pi_theta}[ sum_{t=0}^{T} grad_theta log pi_theta(a_t|s_t) * A^{pi_theta}(s_t,a_t) ]
|
||||
```
|
||||
|
||||
Stochastic gradient ascent on policy performance:
|
||||
|
||||
```
|
||||
theta_{k+1} = theta_k + alpha * grad_theta J(pi_theta_k)
|
||||
```
|
||||
|
||||
**The notation mismatch worth knowing about:** `J(pi_theta)` here denotes the expected
|
||||
**finite-horizon undiscounted** return, but policy gradient implementations typically compute
|
||||
**advantage estimates based on the infinite-horizon discounted return** — despite otherwise
|
||||
using the finite-horizon undiscounted policy gradient formula. This is the ch7 blurring of
|
||||
the two return formulations, appearing in production code.
|
||||
|
||||
## Frameworks Introduced
|
||||
- **The VPG loop** (the template TRPO and PPO both modify, so learn it once):
|
||||
1. Collect a set of trajectories `D_k` by running `pi_k = pi(theta_k)` in the environment.
|
||||
2. Compute rewards-to-go `Rhat_t`.
|
||||
3. Compute advantage estimates `Ahat_t` (any method of advantage estimation) based on the
|
||||
current value function `V_{phi_k}`.
|
||||
4. Estimate the policy gradient:
|
||||
`ghat_k = (1/|D_k|) * sum_{tau in D_k} sum_t grad_theta log pi_theta(a_t|s_t)|_{theta_k} * Ahat_t`
|
||||
5. Update the policy by standard gradient ascent `theta_{k+1} = theta_k + alpha_k * ghat_k`,
|
||||
or another gradient ascent algorithm like Adam.
|
||||
6. Fit the value function by regression on mean-squared error:
|
||||
`phi_{k+1} = argmin_phi (1/(|D_k| T)) * sum sum (V_phi(s_t) - Rhat_t)^2`, typically via
|
||||
gradient descent.
|
||||
|
||||
## Exploration vs. Exploitation
|
||||
VPG trains a **stochastic policy in an on-policy way**, so it explores by sampling actions
|
||||
from the latest version of that policy. How random that is depends on both initial conditions
|
||||
and the training procedure. **Over training the policy typically becomes progressively less
|
||||
random**, because the update rule encourages exploiting rewards already found — **which may
|
||||
cause the policy to get trapped in local optima.** (This paragraph is identical for TRPO and
|
||||
PPO; it is a property of the on-policy stochastic-policy family, not of VPG specifically.)
|
||||
|
||||
## Code Examples
|
||||
Loading and using a trained PyTorch model:
|
||||
|
||||
```python
|
||||
ac = torch.load('path/to/model.pt')
|
||||
actions = ac.act(torch.as_tensor(obs, dtype=torch.float32))
|
||||
```
|
||||
|
||||
The TF1 saved graph exposes three keys: `x` (state input placeholder), `pi` (samples an
|
||||
action conditioned on `x`), and `v` (value estimate for states in `x`).
|
||||
|
||||
## Anti-patterns
|
||||
- **Expecting monotonic improvement.** VPG has none; the step size is unconstrained and a
|
||||
bad step can collapse performance. That collapse is what TRPO (ch15) exists to prevent.
|
||||
- **Under-fitting the value function.** Exercise 2.1 (ch12) shows a policy gradient agent
|
||||
with an untrained value function gets stuck early. The advantage estimate is only as good
|
||||
as `V_phi`.
|
||||
- **Reading the loss.** See ch9's worked example.
|
||||
|
||||
## Reference Tables
|
||||
|
||||
| Relevant paper | Why it is on the list |
|
||||
|----------------|----------------------|
|
||||
| Sutton et al. 2000, *Policy Gradient Methods for RL with Function Approximation* | A timeless classic of RL theory; contains references to the earlier work that led to modern policy gradients |
|
||||
| Schulman 2016(a), *Optimizing Expectations* | Chapter 2 is a lucid introduction to policy gradient theory, including pseudocode |
|
||||
| Duan et al. 2016, *Benchmarking Deep RL for Continuous Control* | A clear benchmark paper showing how VPG in the deep RL setting compares with other deep RL algorithms |
|
||||
| Schulman et al. 2016(b), *High Dimensional Continuous Control Using GAE* | Spinning Up's VPG uses GAE for computing the policy gradient |
|
||||
|
||||
## Key Takeaways
|
||||
1. VPG is the advantage-weighted policy gradient plus mean-squared value regression — nothing else.
|
||||
2. The six-step loop is the base template; TRPO changes step 5, PPO changes step 5 differently.
|
||||
3. Discrete or continuous, MPI-parallelizable, both backends available.
|
||||
4. Its weakness is step-size safety: nothing stops a single update from collapsing the policy.
|
||||
5. Exploration decays as a side effect of exploitation, with local optima the known risk.
|
||||
|
||||
## Connects To
|
||||
- **Ch 9**: the derivation that produces this exact gradient, and GAE.
|
||||
- **Ch 15**: TRPO, which constrains the step this chapter leaves unconstrained.
|
||||
- **Ch 12**: Exercise 2.1, on what happens when `V_phi` is not fit.
|
||||
- **Ch 13**: the benchmark caveat — VPG is educational-grade, not research-grade.
|
||||
|
|
@ -0,0 +1,122 @@
|
|||
# Chapter 15: Trust Region Policy Optimization (TRPO)
|
||||
|
||||
## Core Idea
|
||||
Take the **largest step possible to improve performance** while satisfying a constraint on
|
||||
how close the new and old policies are — measured in **KL-divergence between policies**, not
|
||||
distance in parameter space, because seemingly small parameter differences can produce very
|
||||
large performance differences.
|
||||
|
||||
## Quick Facts
|
||||
- **On-policy.**
|
||||
- Works with **discrete or continuous** action spaces.
|
||||
- **Supports MPI parallelization.**
|
||||
- **Tensorflow only** in Spinning Up; there is no PyTorch TRPO.
|
||||
|
||||
## Key Equations
|
||||
The theoretical update — maximize surrogate advantage subject to a KL trust region:
|
||||
|
||||
```
|
||||
theta_{k+1} = argmax_theta L(theta_k, theta)
|
||||
s.t. Dbar_KL(theta || theta_k) <= delta
|
||||
```
|
||||
|
||||
The **surrogate advantage** measures how `pi_theta` performs relative to the old policy
|
||||
*using data from the old policy* (an importance-weighted advantage):
|
||||
|
||||
```
|
||||
L(theta_k, theta) = E_{s,a ~ pi_theta_k}[ (pi_theta(a|s) / pi_theta_k(a|s)) * A^{pi_theta_k}(s,a) ]
|
||||
|
||||
Dbar_KL(theta || theta_k) = E_{s ~ pi_theta_k}[ D_KL( pi_theta(.|s) || pi_theta_k(.|s) ) ]
|
||||
```
|
||||
|
||||
Both the objective and the constraint are **zero at `theta = theta_k`**, and the gradient of
|
||||
the constraint is **also zero there**.
|
||||
|
||||
## Frameworks Introduced
|
||||
- **Taylor-expand to make it solvable.** Expand objective and constraint to leading order
|
||||
around `theta_k`:
|
||||
```
|
||||
L(theta_k, theta) ~= g^T (theta - theta_k)
|
||||
Dbar_KL(theta||theta_k) ~= 0.5 * (theta - theta_k)^T H (theta - theta_k)
|
||||
```
|
||||
**By happy coincidence, `g` — the gradient of the surrogate advantage at `theta_k` — is
|
||||
exactly equal to the policy gradient `grad_theta J(pi_theta)`.** Lagrangian duality then
|
||||
solves the approximate problem analytically:
|
||||
```
|
||||
theta_{k+1} = theta_k + sqrt( 2*delta / (g^T H^{-1} g) ) * H^{-1} g
|
||||
```
|
||||
**Stopping here would be exactly the Natural Policy Gradient.**
|
||||
|
||||
- **Backtracking line search — the fix that makes it TRPO.** Because the Taylor expansion
|
||||
introduces approximation error, that update may violate the KL constraint or fail to improve
|
||||
the surrogate advantage. So:
|
||||
```
|
||||
theta_{k+1} = theta_k + alpha^j * sqrt( 2*delta / (g^T H^{-1} g) ) * H^{-1} g
|
||||
```
|
||||
where `alpha` in (0,1) is the **backtracking coefficient** and `j` is the **smallest
|
||||
nonnegative integer** such that the new policy satisfies the KL constraint and produces a
|
||||
positive surrogate advantage.
|
||||
- When to use this pattern generally: whenever you solve an approximated problem and can
|
||||
cheaply check the exact condition, shrink the step until the exact condition holds.
|
||||
|
||||
- **Conjugate gradient instead of a matrix inverse.** Computing and storing `H^{-1}` is
|
||||
painfully expensive for networks with thousands or millions of parameters. TRPO solves
|
||||
`Hx = g` for `x = H^{-1} g` with conjugate gradient, which needs only a function computing
|
||||
the matrix-vector product `Hx`, never `H` itself:
|
||||
```
|
||||
Hx = grad_theta( (grad_theta Dbar_KL(theta || theta_k))^T x )
|
||||
```
|
||||
|
||||
## Reference Tables
|
||||
|
||||
| Hyperparameter | Role |
|
||||
|----------------|------|
|
||||
| `delta` | KL-divergence limit — the size of the trust region |
|
||||
| `alpha` | Backtracking coefficient, in (0,1) |
|
||||
| `K` | Maximum number of backtracking steps; `j` ranges over {0, 1, ..., K} |
|
||||
|
||||
The full loop is VPG's six steps with steps 5 replaced by: (a) conjugate gradient to compute
|
||||
`xhat_k ~= Hhat_k^{-1} ghat_k`, where `Hhat_k` is the Hessian of the sample average
|
||||
KL-divergence, then (b) the backtracking line search using
|
||||
`theta_{k+1} = theta_k + alpha^j * sqrt(2 delta / (xhat_k^T Hhat_k xhat_k)) * xhat_k`.
|
||||
|
||||
## Mental Models
|
||||
- **Trust region in policy space, not parameter space.** This is the whole insight. Ordinary
|
||||
policy gradient methods keep policies close in *parameter* space, but small parameter
|
||||
differences can mean very large performance differences — **so a single bad step can
|
||||
collapse the policy performance.** That is what makes large step sizes dangerous with
|
||||
vanilla policy gradients, and it is why VPG's sample efficiency suffers: it has to take
|
||||
small steps.
|
||||
- **TRPO buys back sample efficiency by making large steps safe**, and tends to improve
|
||||
performance **quickly and monotonically**.
|
||||
- Read the algorithm as **three nested approximations** — surrogate objective, Taylor
|
||||
expansion, conjugate gradient — each with a guard, the last being the line search.
|
||||
|
||||
## Exploration vs. Exploitation
|
||||
Identical to VPG: stochastic policy, on-policy sampling, randomness decaying over training as
|
||||
the update rule pushes toward exploiting known rewards, with local optima as the risk.
|
||||
|
||||
## Anti-patterns
|
||||
- **Skipping the line search** and shipping the analytic solution. That is Natural Policy
|
||||
Gradient, and the approximation error it inherits may violate the constraint or fail to
|
||||
improve anything.
|
||||
- **Forming `H` explicitly.** The matrix-vector product formulation exists because the matrix
|
||||
does not fit.
|
||||
- **Reaching for TRPO in PyTorch inside Spinning Up.** It does not exist; use PPO.
|
||||
|
||||
## Key Takeaways
|
||||
1. Constrain the *policy* change in KL, not the *parameter* change — the core contribution.
|
||||
2. Surrogate advantage is importance-weighted old-policy data; both it and the KL constraint
|
||||
vanish at `theta_k`.
|
||||
3. `g` (surrogate advantage gradient) equals the policy gradient — that is why VPG's machinery
|
||||
is reusable here.
|
||||
4. Analytic solution = Natural Policy Gradient; TRPO adds a backtracking line search on top.
|
||||
5. Conjugate gradient replaces the intractable `H^{-1}`.
|
||||
6. Three relevant papers: Schulman 2015 (original), Schulman 2016 (GAE, used here), Kakade and
|
||||
Langford 2002 (the theory motivating and connecting to TRPO's foundations).
|
||||
|
||||
## Connects To
|
||||
- **Ch 14**: the VPG loop TRPO modifies, and the collapse risk it removes.
|
||||
- **Ch 16**: PPO, which pursues the same goal with first-order methods.
|
||||
- **Ch 12**: Exercise 2.1 uses TRPO to demonstrate the value-fitting failure mode.
|
||||
- **Ch 10**: "monotonic improvement theory" — the optional math background this rests on.
|
||||
|
|
@ -0,0 +1,123 @@
|
|||
# Chapter 16: Proximal Policy Optimization (PPO)
|
||||
|
||||
## Core Idea
|
||||
PPO answers the same question as TRPO — how do you take the biggest possible improvement
|
||||
step without stepping so far that you cause performance collapse — but with **first-order
|
||||
methods and a few tricks** instead of a complex second-order method. It is significantly
|
||||
simpler to implement and empirically seems to perform at least as well.
|
||||
|
||||
## Quick Facts
|
||||
- **On-policy.**
|
||||
- Works with **discrete or continuous** action spaces.
|
||||
- **Supports MPI parallelization.**
|
||||
- Two variants exist; Spinning Up implements and documents **PPO-Clip**, the primary variant
|
||||
used at OpenAI.
|
||||
|
||||
## Frameworks Introduced
|
||||
- **The two PPO variants:**
|
||||
- **PPO-Penalty** approximately solves a KL-constrained update like TRPO, but **penalizes
|
||||
the KL-divergence in the objective** instead of making it a hard constraint, and
|
||||
**automatically adjusts the penalty coefficient** over training so it stays appropriately
|
||||
scaled.
|
||||
- **PPO-Clip** has **no KL term in the objective and no constraint at all.** It relies on
|
||||
specialized **clipping in the objective function** to remove the incentive for the new
|
||||
policy to get far from the old.
|
||||
|
||||
- **The PPO-Clip objective.** Update by
|
||||
`theta_{k+1} = argmax_theta E_{s,a ~ pi_theta_k}[ L(s,a,theta_k,theta) ]`, typically with
|
||||
**multiple steps of (usually minibatch) SGD**. The published form:
|
||||
```
|
||||
L = min( ratio * A, clip(ratio, 1-eps, 1+eps) * A )
|
||||
where ratio = pi_theta(a|s) / pi_theta_k(a|s)
|
||||
```
|
||||
and `eps` is a small hyperparameter roughly saying how far the new policy may go.
|
||||
|
||||
- **The simplified equivalent form** (this is the version Spinning Up actually implements,
|
||||
and it is much easier to reason about):
|
||||
```
|
||||
L = min( ratio * A, g(eps, A) )
|
||||
where g(eps, A) = (1 + eps) * A if A >= 0
|
||||
(1 - eps) * A if A < 0
|
||||
```
|
||||
|
||||
## Worked Example
|
||||
**Why clipping removes the incentive to move far — the two cases.**
|
||||
|
||||
**Advantage positive.** The term reduces to `min(ratio, 1+eps) * A`. Because `A > 0`, the
|
||||
objective increases as the action becomes more likely, i.e. as `pi_theta(a|s)` increases.
|
||||
But the `min` caps how much. Once `pi_theta(a|s) > (1+eps) * pi_theta_k(a|s)`, the min kicks
|
||||
in and the term hits a ceiling of `(1+eps) * A`. **The new policy does not benefit by going
|
||||
far away from the old policy.**
|
||||
|
||||
**Advantage negative.** The term reduces to `max(ratio, 1-eps) * A`. Because `A < 0`, the
|
||||
objective increases as the action becomes *less* likely, i.e. as `pi_theta(a|s)` decreases.
|
||||
The `max` caps how much: once `pi_theta(a|s) < (1-eps) * pi_theta_k(a|s)`, it hits a ceiling
|
||||
of `(1-eps) * A`. **Again: the new policy does not benefit by going far away from the old.**
|
||||
|
||||
So **clipping serves as a regularizer by removing incentives for the policy to change
|
||||
dramatically**, and `eps` corresponds to how far the new policy can go while still profiting
|
||||
from the objective.
|
||||
|
||||
**The honest caveat, and Spinning Up's answer.** Clipping goes a long way toward reasonable
|
||||
updates, but **it is still possible to end up with a new policy too far from the old**, and
|
||||
different PPO implementations use a variety of tricks to stave this off. Spinning Up uses a
|
||||
particularly simple one: **early stopping — if the mean KL-divergence of the new policy from
|
||||
the old grows beyond a threshold, stop taking gradient steps.** Note that this reintroduces a
|
||||
KL measurement into an algorithm defined by not having one; the clip is the incentive
|
||||
mechanism, the KL check is the safety net.
|
||||
|
||||
## Reference Tables
|
||||
|
||||
The PPO-Clip loop, as a diff against VPG's six steps (ch14):
|
||||
|
||||
| Step | VPG | PPO-Clip |
|
||||
|------|-----|----------|
|
||||
| 1-3 | Collect trajectories, rewards-to-go, advantage estimates | identical |
|
||||
| 4-5 | Estimate `ghat_k`, single ascent step | **Maximize the PPO-Clip objective, typically via multiple steps of stochastic gradient ascent with Adam** |
|
||||
| 6 | Fit `V_phi` by MSE regression on `Rhat_t` | identical |
|
||||
|
||||
The key structural difference from VPG and TRPO: **multiple gradient steps per batch of data**,
|
||||
which is where the sample-efficiency gain comes from and why the policy needs restraining at all.
|
||||
|
||||
| Relevant paper | Why it is on the list |
|
||||
|----------------|----------------------|
|
||||
| Schulman et al. 2017, *Proximal Policy Optimization Algorithms* | The original PPO paper |
|
||||
| Schulman et al. 2016, *High Dimensional Continuous Control Using GAE* | Spinning Up's PPO uses GAE for the policy gradient |
|
||||
| Heess et al. 2017, *Emergence of Locomotion Behaviours in Rich Environments* | Large-scale empirical analysis of behaviors learned by PPO agents in complex environments — though it uses PPO-Penalty, not PPO-Clip |
|
||||
|
||||
## Exploration vs. Exploitation
|
||||
Identical to VPG and TRPO: stochastic policy, on-policy sampling, decaying randomness, local
|
||||
optima as the failure mode.
|
||||
|
||||
## Mental Models
|
||||
- **PPO trades a hard guarantee for implementability.** TRPO enforces the trust region;
|
||||
PPO removes the *incentive* to leave it. That is a weaker statement, and PPO's practical
|
||||
dominance says the weaker statement is usually enough.
|
||||
- **The clip is one-sided per case.** It only ever caps the *beneficial* direction. It never
|
||||
penalizes moving the wrong way — it just stops paying you for moving further the right way.
|
||||
- Careful with terminology: **ch9 notes PPO is often called a policy gradient algorithm,
|
||||
though this is slightly inaccurate** — it maximizes a surrogate objective rather than
|
||||
following `grad J` directly.
|
||||
|
||||
## Anti-patterns
|
||||
- **Implementing PPO-Clip and calling the result PPO without saying which variant.** Two
|
||||
public implementations named "PPO" (ModularRL, rllab) implement PPO-**Penalty**; comparing
|
||||
against them without noticing is a real reproduction hazard the docs explicitly flag.
|
||||
- **Relying on the clip alone at large step counts.** More SGD steps per batch means more
|
||||
opportunity to drift; that is precisely why the early-stopping KL check exists.
|
||||
- **Tuning `eps` as if it were a learning rate.** It bounds policy change per update; the
|
||||
number of SGD steps and the learning rate interact with it.
|
||||
|
||||
## Key Takeaways
|
||||
1. Same goal as TRPO, first-order machinery: simpler to implement, at least as good empirically.
|
||||
2. PPO-Clip (no KL term, no constraint) is the OpenAI default and Spinning Up's implementation.
|
||||
3. The simplified `g(eps, A)` form makes the mechanism obvious: a ceiling in the profitable
|
||||
direction, in both the positive- and negative-advantage cases.
|
||||
4. Clipping is an incentive change, not a guarantee — Spinning Up backs it with KL early stopping.
|
||||
5. Multiple minibatch SGD steps per batch is the structural difference from VPG and TRPO.
|
||||
|
||||
## Connects To
|
||||
- **Ch 15**: TRPO's hard constraint, the thing PPO replaces.
|
||||
- **Ch 9**: the surrogate-objective framing and why "policy gradient algorithm" is imprecise here.
|
||||
- **Ch 12**: Exercise 1.2 asks you to implement the MLP diagonal Gaussian policy for PPO.
|
||||
- **Ch 13**: use OpenAI Baselines' PPO, not this one, for research comparisons.
|
||||
|
|
@ -0,0 +1,122 @@
|
|||
# Chapter 17: Deep Deterministic Policy Gradient (DDPG)
|
||||
|
||||
## Core Idea
|
||||
DDPG concurrently learns a Q-function and a policy: it uses off-policy data and the Bellman
|
||||
equation to learn the Q-function, and uses the Q-function to learn the policy. It exists
|
||||
because `max_a Q*(s,a)` — trivial over discrete actions — is intractable over continuous ones,
|
||||
and DDPG replaces that max with a learned, differentiable approximation.
|
||||
|
||||
## Quick Facts
|
||||
- **Off-policy.**
|
||||
- **Continuous action spaces only.**
|
||||
- Can be thought of as **deep Q-learning for continuous action spaces.**
|
||||
- The Spinning Up implementation **does not support parallelization.**
|
||||
|
||||
## Frameworks Introduced
|
||||
- **The continuous-argmax problem, and the substitution that solves it.** With finitely many
|
||||
discrete actions, the max poses no problem — compute Q for each and compare, which also
|
||||
immediately hands you the maximizing action. With a continuous action space you cannot
|
||||
exhaustively evaluate, and a general optimization subroutine would have to run **every time
|
||||
the agent wants to take an action**, which is unacceptable. But because the action space is
|
||||
continuous, `Q*(s,a)` is presumed **differentiable with respect to the action** — so you can
|
||||
learn a policy `mu(s)` by gradient ascent and approximate
|
||||
`max_a Q(s,a) ~= Q(s, mu(s))`.
|
||||
- When to use this pattern: any time an inner argmax over a continuous variable sits inside
|
||||
an outer loop. Amortize it into a learned function.
|
||||
|
||||
- **Mean-Squared Bellman Error (MSBE)** — the loss that all deep Q-learning minimizes:
|
||||
```
|
||||
L(phi, D) = E_{(s,a,r,s',d) ~ D}[ ( Q_phi(s,a) - ( r + gamma*(1-d)*max_a' Q_phi(s',a') ) )^2 ]
|
||||
```
|
||||
`d` indicates whether `s'` is terminal, using the Python convention of `True` as 1: when
|
||||
`d == True` the Q-function should show the agent gets no additional reward after this state.
|
||||
|
||||
- **Trick One: Replay Buffers.** All standard algorithms training a deep network to approximate
|
||||
`Q*(s,a)` use an experience replay buffer — the set `D` of previous experiences.
|
||||
- **The buffer size is a real tuning decision, in both directions:** it should be large
|
||||
enough to contain a wide range of experiences, but keeping everything may not be good
|
||||
either. **Use only the very-most recent data and you overfit to it and things break; use
|
||||
too much experience and you may slow down learning.**
|
||||
- **Why old data is admissible at all** — this is the cleanest statement of what off-policy
|
||||
means: **the Bellman equation does not care which transition tuples are used, how the
|
||||
actions were selected, or what happens after a given transition**, because the optimal
|
||||
Q-function must satisfy it for *all* possible transitions. So any transition you have ever
|
||||
experienced is fair game for MSBE minimization.
|
||||
|
||||
- **Trick Two: Target Networks.** The target `r + gamma*(1-d)*max_a' Q_phi(s',a')` depends on
|
||||
the same parameters `phi` being trained, which makes MSBE minimization unstable. The fix is
|
||||
a second network that lags the first, with parameters `phi_targ`.
|
||||
- **DQN-style**: copy the main network into the target every fixed number of steps.
|
||||
- **DDPG-style**: **polyak averaging**, once per main network update:
|
||||
`phi_targ <- rho * phi_targ + (1-rho) * phi`, with `rho` in (0,1), usually close to 1.
|
||||
(`rho` is called `polyak` in the code.)
|
||||
|
||||
- **DDPG detail: the max in the target.** Since the max over continuous actions is the original
|
||||
problem, DDPG uses a **target policy network** `mu_theta_targ` to compute an action that
|
||||
approximately maximizes `Q_phi_targ`, found the same way as the target Q-function — by
|
||||
polyak averaging the policy parameters over training. Final Q-loss:
|
||||
```
|
||||
L(phi,D) = E[ ( Q_phi(s,a) - ( r + gamma*(1-d)*Q_phi_targ(s', mu_theta_targ(s')) ) )^2 ]
|
||||
```
|
||||
|
||||
- **Policy learning is one line**: maximize `E_{s~D}[ Q_phi(s, mu_theta(s)) ]` by gradient
|
||||
ascent **with respect to policy parameters only** — the Q-function parameters are treated
|
||||
as constants.
|
||||
|
||||
## Exploration vs. Exploitation
|
||||
DDPG trains a **deterministic policy off-policy**. Because the policy is deterministic,
|
||||
on-policy exploration would probably not try a wide enough variety of actions early on to
|
||||
find useful learning signal. So **noise is added to actions at training time**.
|
||||
- The original DDPG paper recommended **time-correlated OU noise**, but **more recent results
|
||||
suggest uncorrelated, mean-zero Gaussian noise works perfectly well** — and since it is
|
||||
simpler, it is preferred.
|
||||
- You may reduce the noise scale over training to get higher-quality data. **Spinning Up does
|
||||
not do this and keeps the noise scale fixed throughout.**
|
||||
- **At test time, no noise is added** — that is how you see how well the policy exploits what
|
||||
it has learned.
|
||||
- **`start_steps` trick**: for a fixed number of steps at the beginning, the agent takes
|
||||
actions sampled from a **uniform random distribution over valid actions**, then reverts to
|
||||
normal DDPG exploration.
|
||||
|
||||
## Reference Tables
|
||||
|
||||
The DDPG loop:
|
||||
|
||||
| Step | Action |
|
||||
|------|--------|
|
||||
| Init | Set target parameters equal to main: `theta_targ <- theta`, `phi_targ <- phi` |
|
||||
| Act | Observe `s`, select `a = clip(mu_theta(s) + epsilon, a_Low, a_High)`, `epsilon ~ N` |
|
||||
| Store | Execute `a`, observe `s'`, `r`, `d`; store `(s,a,r,s',d)` in `D`; reset if terminal |
|
||||
| Update | Sample a batch `B` from `D`; compute targets `y = r + gamma*(1-d)*Q_phi_targ(s', mu_theta_targ(s'))` |
|
||||
| | One gradient descent step on `(Q_phi(s,a) - y)^2` averaged over `B` |
|
||||
| | One gradient ascent step on `Q_phi(s, mu_theta(s))` averaged over `B` |
|
||||
| | Polyak-update both target networks |
|
||||
|
||||
| Relevant paper | Why it is on the list |
|
||||
|----------------|----------------------|
|
||||
| Silver et al. 2014, *Deterministic Policy Gradient Algorithms* | Establishes the theory underlying deterministic policy gradients (DPG) |
|
||||
| Lillicrap et al. 2016, *Continuous Control With Deep RL* | Adapts the theoretically-grounded DPG algorithm to the deep RL setting, giving DDPG |
|
||||
|
||||
## Anti-patterns
|
||||
- **DDPG on a discrete action space.** The entire design is the continuous-action workaround.
|
||||
- **Trusting the original paper's full recipe.** Ch10 uses DDPG as its example: the paper
|
||||
suggests a complex architecture, an initialization scheme and batch normalization, none of
|
||||
which are strictly necessary, and some of the best DDPG results use simpler networks.
|
||||
- **Assuming DDPG is stable.** It is **frequently brittle with respect to hyperparameters and
|
||||
other kinds of tuning** — that is the premise of ch18.
|
||||
- **Shape bugs in the Q-function.** Exercise 2.2 (ch12) plants exactly one in DDPG's critic
|
||||
and it silently ruins learning through broadcasting.
|
||||
|
||||
## Key Takeaways
|
||||
1. `Q(s, mu(s))` replaces `max_a Q(s,a)` — the whole reason DDPG exists.
|
||||
2. Replay buffers are licensed by the Bellman equation's indifference to how data was collected.
|
||||
3. Target networks fix the moving-target instability; DDPG uses polyak averaging, DQN uses copies.
|
||||
4. Two target networks in DDPG: target Q *and* target policy.
|
||||
5. Explore with additive Gaussian action noise plus uniform-random `start_steps`; no noise at test.
|
||||
6. Replay buffer size is a genuine trade-off, wrong in both directions.
|
||||
|
||||
## Connects To
|
||||
- **Ch 7**: `a*(s) = argmax_a Q*(s,a)` and the Bellman optimality equation this is built on.
|
||||
- **Ch 8**: DDPG as the named interpolation between policy optimization and Q-learning.
|
||||
- **Ch 18**: TD3, the three tricks that fix DDPG's brittleness.
|
||||
- **Ch 12**: Exercise 2.2, the silent bug planted in this algorithm.
|
||||
|
|
@ -0,0 +1,107 @@
|
|||
# Chapter 18: Twin Delayed DDPG (TD3)
|
||||
|
||||
## Core Idea
|
||||
DDPG's common failure mode is that **the learned Q-function begins to dramatically
|
||||
overestimate Q-values, which then breaks the policy, because the policy exploits the errors
|
||||
in the Q-function.** TD3 fixes this with three tricks, and nothing else changes.
|
||||
|
||||
## Quick Facts
|
||||
- **Off-policy.**
|
||||
- **Continuous action spaces only.**
|
||||
- The Spinning Up implementation **does not support parallelization.**
|
||||
|
||||
## Frameworks Introduced
|
||||
- **Trick One: Clipped Double-Q Learning.** Learn **two** Q-functions instead of one (hence
|
||||
"twin") and **use the smaller of the two Q-values to form the targets** in the Bellman error
|
||||
loss functions. Both are then regressed toward that single shared target:
|
||||
```
|
||||
y(r,s',d) = r + gamma*(1-d) * min_{i=1,2} Q_{phi_i,targ}(s', a'(s'))
|
||||
L(phi_1, D) = E[ (Q_phi_1(s,a) - y)^2 ]
|
||||
L(phi_2, D) = E[ (Q_phi_2(s,a) - y)^2 ]
|
||||
```
|
||||
**Using the smaller Q-value for the target, and regressing toward that, helps fend off
|
||||
overestimation** — a pessimistic estimate cannot be exploited the way an optimistic one can.
|
||||
|
||||
- **Trick Two: "Delayed" Policy Updates.** Update the policy (and the target networks) **less
|
||||
frequently than the Q-function**. **The paper recommends one policy update for every two
|
||||
Q-function updates.** This **damps the volatility that normally arises in DDPG because of
|
||||
how a policy update changes the target.** Controlled by `policy_delay`.
|
||||
|
||||
- **Trick Three: Target Policy Smoothing.** Add **clipped** noise to each dimension of the
|
||||
target action, then clip the result back into the valid action range:
|
||||
```
|
||||
a'(s') = clip( mu_theta_targ(s') + clip(epsilon, -c, c), a_Low, a_High ), epsilon ~ N(0, sigma)
|
||||
```
|
||||
**This is a regularizer.** It addresses a specific DDPG failure: **if the Q-function
|
||||
approximator develops an incorrect sharp peak for some actions, the policy will quickly
|
||||
exploit that peak and then have brittle or incorrect behavior.** Smoothing Q over similar
|
||||
actions averts it.
|
||||
|
||||
- **Policy learning is unchanged from DDPG except in which critic it uses:** maximize
|
||||
`E_{s~D}[ Q_phi_1(s, mu_theta(s)) ]` — **just the first Q-function.** (SAC differs here; see
|
||||
ch19.)
|
||||
|
||||
## Reference Tables
|
||||
|
||||
| Trick | Fixes | Mechanism |
|
||||
|-------|-------|-----------|
|
||||
| Clipped double-Q | Q-value overestimation | Two critics, `min` of the two forms the shared target |
|
||||
| Delayed policy updates | Volatility from policy updates moving the target | One policy + target update per `policy_delay` critic updates |
|
||||
| Target policy smoothing | Policy exploiting sharp incorrect peaks in Q | Clipped Gaussian noise on the target action, then clip to action bounds |
|
||||
|
||||
| Hyperparameter | Role |
|
||||
|----------------|------|
|
||||
| `sigma` | Std dev of the target-smoothing noise |
|
||||
| `c` | Clip bound on that noise |
|
||||
| `policy_delay` | Q-updates per policy update; paper recommends 2 |
|
||||
| `rho` (`polyak`) | Target network averaging coefficient, inherited from DDPG |
|
||||
| `start_steps` | Initial uniform-random action steps, inherited from DDPG |
|
||||
|
||||
The loop is DDPG's, with two changes inside the update block: compute the smoothed target
|
||||
action `a'(s')` first, update **both** Q-functions toward the clipped-double-Q target, and
|
||||
then — **only when `j mod policy_delay == 0`** — take the policy ascent step and polyak-update
|
||||
all three target networks.
|
||||
|
||||
## Exploration vs. Exploitation
|
||||
Identical to DDPG: deterministic policy explored with **uncorrelated mean-zero Gaussian action
|
||||
noise** at training time, optionally decayed (Spinning Up keeps it fixed), **no noise at test
|
||||
time**, and a `start_steps` phase of uniform random actions at the beginning.
|
||||
|
||||
Note the distinction that is easy to blur: **exploration noise is added to the acting policy;
|
||||
target policy smoothing noise is added to the target action inside the update.** Two different
|
||||
noises, two different jobs.
|
||||
|
||||
## Mental Models
|
||||
- **All three tricks are forms of pessimism or patience.** Take the smaller estimate, wait
|
||||
longer before acting on the critic, and refuse to believe a Q-value that does not hold up
|
||||
under a small action perturbation. Overestimation is the disease; conservatism is the cure.
|
||||
- **Think of the policy as an adversary against your critic's errors.** Anything the critic
|
||||
gets wrong in the optimistic direction is exactly what the policy will find. This is the
|
||||
same failure shape as model bias in ch8's model-based methods.
|
||||
- **"Together, these three tricks result in substantially improved performance over baseline
|
||||
DDPG"** — they are presented as a package, and the exercise (ch12, 1.3) has you implement
|
||||
the losses that contain all three.
|
||||
|
||||
## Anti-patterns
|
||||
- **Implementing only clipped double-Q and calling it TD3.** The three tricks address three
|
||||
distinct failure modes and the paper's result is for the package.
|
||||
- **Using `min` over the two critics for the *policy* loss.** TD3 uses `Q_phi_1` alone for the
|
||||
policy; using the min there is a SAC choice, not a TD3 one.
|
||||
- **Updating the target networks on every critic step.** The target updates are inside the
|
||||
`policy_delay` branch, along with the policy update.
|
||||
|
||||
## Key Takeaways
|
||||
1. TD3 = DDPG + clipped double-Q + delayed policy updates + target policy smoothing.
|
||||
2. The disease being treated is Q-value overestimation, which the policy actively exploits.
|
||||
3. `min` of two critics forms the target; the policy maximizes `Q_phi_1` only.
|
||||
4. Policy and target-network updates both live behind `policy_delay` (recommended 2).
|
||||
5. Target smoothing noise is clipped twice: clip the noise to `[-c, c]`, then clip the
|
||||
resulting action to the valid range.
|
||||
6. Relevant paper: Fujimoto et al, 2018, *Addressing Function Approximation Error in
|
||||
Actor-Critic Methods.*
|
||||
|
||||
## Connects To
|
||||
- **Ch 17**: DDPG, whose brittleness is TD3's entire premise.
|
||||
- **Ch 19**: SAC, which borrows clipped double-Q and gets smoothing for free from stochasticity.
|
||||
- **Ch 12**: Exercise 1.3 asks you to write exactly these losses.
|
||||
- **Ch 13**: TD3 is one of the three research-grade Spinning Up implementations.
|
||||
|
|
@ -0,0 +1,133 @@
|
|||
# Chapter 19: Soft Actor-Critic (SAC)
|
||||
|
||||
## Core Idea
|
||||
SAC optimizes a **stochastic** policy in an **off-policy** way, forming a bridge between
|
||||
stochastic policy optimization and DDPG-style approaches. Its central feature is **entropy
|
||||
regularization**: the policy is trained to maximize a trade-off between expected return and
|
||||
the randomness of the policy.
|
||||
|
||||
## Quick Facts
|
||||
- **Off-policy.**
|
||||
- The version implemented here is **continuous action spaces only** — an alternate version,
|
||||
with a slightly changed policy update rule, can handle discrete action spaces.
|
||||
- **Does not support parallelization.**
|
||||
- SAC is **not a direct successor to TD3** — they were published roughly concurrently — but it
|
||||
incorporates the clipped double-Q trick, and its inherent policy stochasticity gives it
|
||||
something like target policy smoothing for free.
|
||||
|
||||
## Frameworks Introduced
|
||||
- **Entropy-regularized RL.** Entropy `H(P) = E_{x~P}[-log P(x)]` says how random a random
|
||||
variable is: a coin that almost always comes up heads has low entropy, a fair coin has high
|
||||
entropy. The agent gets a bonus reward at each timestep proportional to policy entropy,
|
||||
changing the RL problem to:
|
||||
```
|
||||
pi* = argmax_pi E_{tau~pi}[ sum_t gamma^t ( R(s_t,a_t,s_{t+1}) + alpha * H(pi(.|s_t)) ) ]
|
||||
```
|
||||
`alpha > 0` is the trade-off coefficient. **This has a close connection to the
|
||||
exploration-exploitation trade-off: increasing entropy results in more exploration, which
|
||||
can accelerate learning later on, and can prevent the policy from prematurely converging to
|
||||
a bad local optimum.**
|
||||
|
||||
- **The modified value functions.** `V^pi` includes the entropy bonus from **every** timestep;
|
||||
`Q^pi` includes it from every timestep **except the first**. They connect by
|
||||
```
|
||||
V^pi(s) = E_{a~pi}[ Q^pi(s,a) ] + alpha * H(pi(.|s))
|
||||
= E_{a~pi}[ Q^pi(s,a) - alpha * log pi(a|s) ]
|
||||
```
|
||||
**This setup is a little bit arbitrary** — you could instead have `Q^pi` include the first
|
||||
timestep's bonus — and **the choice of definition varies slightly across papers on the
|
||||
subject.** Check before comparing equations across sources.
|
||||
|
||||
- **What SAC keeps from TD3, and what it changes.** SAC learns a policy `pi_theta` and two
|
||||
Q-functions `Q_phi_1`, `Q_phi_2`.
|
||||
- **Same as TD3**: both Q-functions learned by MSBE minimization regressing to a **single
|
||||
shared target**; the target computed using **target Q-networks obtained by polyak
|
||||
averaging**; the **clipped double-Q trick**.
|
||||
- **Different from TD3**: (1) the target includes an **entropy regularization term**;
|
||||
(2) the **next-state actions in the target come from the current policy, not a target
|
||||
policy**; (3) **no explicit target policy smoothing** — TD3 trains a deterministic policy
|
||||
and so needs added noise to smooth, whereas **SAC's policy stochasticity is sufficient to
|
||||
get a similar effect**.
|
||||
|
||||
- **The Q-loss.** Rewrite the entropy-regularized Bellman equation using
|
||||
`H = -log pi`, approximate the expectation with samples, and take the min over the two
|
||||
critics:
|
||||
```
|
||||
y(r,s',d) = r + gamma*(1-d) * ( min_{j=1,2} Q_{phi_j,targ}(s', a'~) - alpha * log pi_theta(a'~|s') )
|
||||
where a'~ ~ pi_theta(.|s')
|
||||
```
|
||||
**Notation matters here:** `r` and `s'` come from the **replay buffer**, but `a'~` must be
|
||||
**sampled fresh from the current policy** — hence the tilde. Mixing these up is a silent bug.
|
||||
|
||||
- **Policy learning via the reparameterization trick.** The policy should maximize `V^pi(s)`,
|
||||
i.e. `E_{a~pi}[ Q^pi(s,a) - alpha log pi(a|s) ]`. The expectation's distribution depends on
|
||||
the policy parameters, which is the pain point; the reparameterization trick rewrites it as
|
||||
an expectation over **noise**, which has no parameter dependence. Using a **squashed Gaussian
|
||||
policy**:
|
||||
```
|
||||
a~_theta(s, xi) = tanh( mu_theta(s) + sigma_theta(s) * xi ), xi ~ N(0, I)
|
||||
```
|
||||
giving the policy objective
|
||||
```
|
||||
max_theta E_{s~D, xi~N}[ min_{j=1,2} Q_phi_j(s, a~_theta(s,xi)) - alpha * log pi_theta(a~_theta(s,xi)|s) ]
|
||||
```
|
||||
**almost the same as DDPG and TD3 policy optimization, except for the min-double-Q trick,
|
||||
the stochasticity, and the entropy term.** Note that unlike TD3 (which uses `Q_phi_1` only),
|
||||
**SAC uses the min of the two approximators in the policy loss too.**
|
||||
|
||||
## Reference Tables
|
||||
|
||||
**Two ways SAC's policy differs from the VPG/TRPO/PPO policies** — both are load-bearing:
|
||||
|
||||
| Difference | SAC | VPG / TRPO / PPO |
|
||||
|-----------|-----|------------------|
|
||||
| **Squashing function** | `tanh` ensures actions are bounded to a finite range. Before the tanh the policy is a factored Gaussian; after it, it is not. Log-probabilities are still computable in closed form (see the paper appendix) | No squashing |
|
||||
| **Std dev parameterization** | log stds are **outputs of the neural network**, so they depend on state in a complex way | log stds are **state-independent parameter vectors** |
|
||||
|
||||
**SAC with state-independent log std devs, in the authors' experience, did not work.**
|
||||
|
||||
| Variant | Status |
|
||||
|---------|--------|
|
||||
| Fixed entropy coefficient `alpha` | What Spinning Up implements, for simplicity |
|
||||
| Entropy-constrained (varies `alpha` over training) | **Generally preferred by practitioners** |
|
||||
| Older SAC that also learns a value function `V_psi` | Superseded; the docs cover the modern version that omits it |
|
||||
|
||||
## Exploration vs. Exploitation
|
||||
SAC trains a stochastic policy with entropy regularization and explores **on-policy**.
|
||||
**`alpha` explicitly controls the explore-exploit trade-off** — higher means more exploration,
|
||||
lower means more exploitation. **The right coefficient may vary from environment to environment
|
||||
and could require careful tuning.** At test time, **remove the stochasticity and use the mean
|
||||
action instead of a sample**; this tends to improve performance over the stochastic policy.
|
||||
(This is why `test_policy` has a SAC-only `--deterministic` flag — see ch5.) `start_steps`
|
||||
uniform-random exploration at the beginning applies here too.
|
||||
|
||||
## Anti-patterns
|
||||
- **Evaluating SAC with the stochastic policy.** The correct evaluation policy is the
|
||||
deterministic mean; `test_policy` defaults to the stochastic one and needs `-d`.
|
||||
- **Using state-independent log stds in SAC** because that is what PPO does. It did not work.
|
||||
- **Sampling the target's next action from the replay buffer.** It must come fresh from the
|
||||
current policy.
|
||||
- **Assuming SAC's `Q^pi` definition matches another paper's.** The placement of the
|
||||
first-timestep entropy bonus is a convention that varies.
|
||||
- **Treating `alpha` as a set-and-forget constant across environments.** It is the
|
||||
explore-exploit dial and is environment-specific; the entropy-constrained variant exists
|
||||
because tuning it by hand is hard.
|
||||
|
||||
## Key Takeaways
|
||||
1. Entropy regularization is the defining feature: maximize return plus `alpha` times policy
|
||||
entropy, which directly controls exploration.
|
||||
2. SAC keeps TD3's clipped double-Q and polyak targets, adds an entropy term to the target,
|
||||
uses the current policy (not a target policy) for next-state actions, and needs no explicit
|
||||
smoothing.
|
||||
3. The squashed Gaussian plus reparameterization trick makes the stochastic policy
|
||||
differentiable end to end.
|
||||
4. SAC uses `min` over both critics in the policy loss; TD3 uses only `Q_phi_1`.
|
||||
5. State-dependent log stds are required — state-independent ones did not work.
|
||||
6. Evaluate with the mean action. Spinning Up ships the fixed-`alpha` variant; practitioners
|
||||
generally prefer the entropy-constrained one.
|
||||
|
||||
## Connects To
|
||||
- **Ch 18**: TD3, whose clipped double-Q trick SAC borrows and whose smoothing it obviates.
|
||||
- **Ch 7**: the diagonal Gaussian policy and the two ways of parameterizing log stds.
|
||||
- **Ch 5**: the SAC-only `--deterministic` flag on `test_policy`.
|
||||
- **Ch 10**: the reparameterization trick is on the required deep-learning background list.
|
||||
|
|
@ -0,0 +1,99 @@
|
|||
# Chapter 20: Logger, MPI Tools and Run Utils
|
||||
|
||||
## Core Idea
|
||||
The three utility modules are the only code shared between algorithms — logging (which is
|
||||
also model saving and config recording), MPI data-parallelism, and ExperimentGrid — and
|
||||
each has a small fixed usage pattern worth memorizing.
|
||||
|
||||
## Frameworks Introduced
|
||||
- **`Logger` and `EpochLogger`.** `Logger` carries the basic functionality: saving
|
||||
diagnostics, hyperparameter configurations, the state of a training run, and the trained
|
||||
model. `EpochLogger` adds a thin layer that makes it easy to track the **average, standard
|
||||
deviation, min and max** of a diagnostic over each epoch **and across MPI workers**.
|
||||
**All Spinning Up algorithm implementations use an EpochLogger.**
|
||||
- The three-call pattern: **`store`** accumulates values into internal state,
|
||||
**`log_tabular`** computes the statistics over everything stored, **`dump_tabular`**
|
||||
writes to file and stdout. **The internal state is wiped clean after `log_tabular`,
|
||||
to prevent leakage into the next epoch's statistics.**
|
||||
|
||||
- **The MPI + PyTorch pattern** — three steps, in this order:
|
||||
1. At the beginning of the training script, call **`setup_pytorch_for_mpi()`**. This is the
|
||||
fix for a real problem: each separate process's PyTorch instance tries to grab too many
|
||||
threads and they clobber each other.
|
||||
2. After constructing a PyTorch module, call **`sync_params(module)`**.
|
||||
3. During gradient descent, call **`mpi_avg_grads`** after the backward pass and before the
|
||||
optimizer step.
|
||||
- The two main ingredients are therefore **syncing parameters** and **averaging gradients
|
||||
before they are used by the adaptive optimizer** — the order matters, because averaging
|
||||
after the optimizer step would give each worker a different adaptive state.
|
||||
|
||||
- **ExperimentGrid** — a tool for hyperparameter ablations, based on but simpler than rllab's
|
||||
VariantGenerator. `eg.add(param_name, values, shorthand, in_name)` then
|
||||
`eg.run(thunk, **run_kwargs)`; `ExperimentGrid.run` uses `call_experiment` to launch the
|
||||
thunk. See ch4 for the full usage.
|
||||
|
||||
## Code Examples
|
||||
The EpochLogger statistic pattern:
|
||||
|
||||
```python
|
||||
from spinup.utils.logx import EpochLogger
|
||||
epoch_logger = EpochLogger()
|
||||
for i in range(10):
|
||||
epoch_logger.store(Test=i)
|
||||
epoch_logger.log_tabular('Test', with_min_and_max=True)
|
||||
epoch_logger.dump_tabular()
|
||||
```
|
||||
|
||||
```
|
||||
-------------------------------------
|
||||
| AverageTest | 4.5 |
|
||||
| StdTest | 2.87 |
|
||||
| MaxTest | 9 |
|
||||
| MinTest | 0 |
|
||||
-------------------------------------
|
||||
```
|
||||
|
||||
The MPI gradient-averaging step, in place:
|
||||
|
||||
```python
|
||||
optimizer.zero_grad()
|
||||
loss = compute_loss(module)
|
||||
loss.backward()
|
||||
mpi_avg_grads(module) # averages gradient buffers across MPI processes
|
||||
optimizer.step()
|
||||
```
|
||||
|
||||
## Reference Tables
|
||||
|
||||
| Module | Contents |
|
||||
|--------|----------|
|
||||
| `spinup.utils.logx` | `Logger`, `EpochLogger` |
|
||||
| `spinup.utils.mpi_tools` | Core MPI utilities |
|
||||
| `spinup.utils.mpi_pytorch` | `setup_pytorch_for_mpi`, `sync_params`, `mpi_avg_grads` |
|
||||
| `spinup.utils.mpi_tf` | AdamOptimizer across MPI processes. **Explicitly "a bit hacky"** — for something more sophisticated and general-purpose, the docs point to horovod |
|
||||
| `spinup.utils.run_utils` | `ExperimentGrid`, `call_experiment`, `setup_logger_kwargs` |
|
||||
| `spinup.utils.plot` | The plotter (documented in ch6) |
|
||||
|
||||
## Anti-patterns
|
||||
- **Calling `mpi_avg_grads` after `optimizer.step()`.** The averaging must happen before the
|
||||
adaptive optimizer consumes the gradients, or workers diverge in optimizer state.
|
||||
- **Skipping `setup_pytorch_for_mpi()`.** The thread-clobbering problem it fixes is a real
|
||||
performance failure, not a theoretical one.
|
||||
- **Expecting `log_tabular` to be idempotent.** It wipes the accumulated state; calling it
|
||||
twice for the same key in one epoch gives you statistics over nothing.
|
||||
- **Reaching for `mpi_tf` as a general distributed-training solution.** The docs say to use
|
||||
horovod instead if you need something serious.
|
||||
|
||||
## Key Takeaways
|
||||
1. `store` / `log_tabular` / `dump_tabular` is the whole logger interface, and the state
|
||||
resets on `log_tabular`.
|
||||
2. `EpochLogger` aggregates across MPI workers as well as across an epoch.
|
||||
3. MPI PyTorch is three calls in a fixed order: setup, sync params, average grads before step.
|
||||
4. The logger is also the model-saving and config-recording path (ch5).
|
||||
5. `mpi_tf` is acknowledged as hacky; horovod is the recommended alternative.
|
||||
|
||||
## Connects To
|
||||
- **Ch 4**: ExperimentGrid usage and the CLI it backs.
|
||||
- **Ch 5**: the outputs the logger writes — `progress.txt`, `config.json`, the save directories.
|
||||
- **Ch 6**: the plotter, which reads what `dump_tabular` writes.
|
||||
- **Ch 2**: OpenMPI, the system dependency all of this rests on.
|
||||
|
|
@ -0,0 +1,87 @@
|
|||
# Cheatsheet
|
||||
|
||||
Decision rules, thresholds and tells from Spinning Up. One page; keep it beside you.
|
||||
|
||||
## Which algorithm?
|
||||
|
||||
| If | Then | Because |
|
||||
|----|------|---------|
|
||||
| Discrete actions | PPO (or VPG to learn) | DDPG, TD3 and shipped SAC are continuous-only |
|
||||
| Continuous, stability matters most | PPO | Directly optimizes performance; on-policy is stable |
|
||||
| Continuous, sample efficiency matters most | SAC, else TD3 | Off-policy data reuse; both are research-grade here |
|
||||
| You are learning the field | VPG first, then DQN, A2C, PPO, DDPG | Simplest first; complexity added gradually |
|
||||
| You need a research baseline for TRPO/PPO | OpenAI Baselines, not Spinning Up | Spinning Up's on-policy trio omits observation normalization and normalized value targets |
|
||||
| You need a research baseline for DDPG/TD3/SAC | Spinning Up is fine | Roughly at parity with best reported results |
|
||||
| Partial observability or pixels | None of these as shipped | All six are non-recurrent MLP actor-critics |
|
||||
|
||||
## Debugging a run that does not learn
|
||||
|
||||
1. **Assume it is a bug, not the hyperparameters.** Bad hyperparameters degrade performance;
|
||||
if yours resemble the paper's, they are probably not the cause.
|
||||
2. **Check tensor shapes.** A `[N]` vs `[N,1]` mismatch is broadcast-compatible, raises
|
||||
nothing, and silently corrupts the Bellman backup into an `[N,N]` matrix.
|
||||
3. **Check what your loss is computed on** — wrong equation, wrong distribution, or data
|
||||
piped to the wrong place are the three usual causes.
|
||||
4. **Do not read the policy-gradient loss as a health signal.** Only average return means
|
||||
anything; the loss can go to negative infinity while performance craters.
|
||||
5. **Instrument more.** Mean/std/min/max of returns, episode lengths and value estimates;
|
||||
objective losses; exploration parameters (policy entropy, epsilon). Watch videos too.
|
||||
6. **Test in more than one environment** once results look promising — code can work in one
|
||||
environment despite a breaking bug.
|
||||
|
||||
## Thresholds and defaults
|
||||
|
||||
| Quantity | Value | Source |
|
||||
|----------|-------|--------|
|
||||
| Debug-stage turnaround target | **under 5 minutes** locally | Ch 10 |
|
||||
| Random seeds, minimum | **3** | Ch 10 |
|
||||
| Random seeds, thorough | **10 or more** | Ch 10, Ch 13 |
|
||||
| Debug environments | CartPole-v0, InvertedPendulum-v0, FrozenLake-v0, HalfCheetah-v2 at 100-250 steps | Ch 10 |
|
||||
| From-scratch implementation size | ~250-300 lines; no-frills VPG ~80 | Ch 10 |
|
||||
| On-policy benchmark network | (64, 32), tanh | Ch 13 |
|
||||
| Off-policy benchmark network | (256, 256), relu | Ch 13 |
|
||||
| On-policy batch | 4000 env steps per update | Ch 13 |
|
||||
| Off-policy minibatch | 100 per gradient step | Ch 13 |
|
||||
| TD3 `policy_delay` | 2 | Ch 18 |
|
||||
| Polyak `rho` | in (0,1), usually close to 1 | Ch 17 |
|
||||
| Benchmark length | 3M timesteps, 5 MuJoCo envs | Ch 13 |
|
||||
| Plot smoothing used in the book | 11-epoch window | Ch 13 |
|
||||
|
||||
## Choosing a research frame
|
||||
|
||||
| Frame | Scope | Wraps up in | Main risk |
|
||||
|-------|-------|-------------|-----------|
|
||||
| Improve an existing approach | Narrow | A few months | Tweaks fail and you have no signal on what next |
|
||||
| Unsolved benchmark | Broad | Months to a year-plus | May need a breakthrough; long time, no progress |
|
||||
| New problem setting | Open-ended | Unbounded | You must design the benchmark too; cannot go looking for these |
|
||||
|
||||
## Rigor checklist before reporting
|
||||
|
||||
- [ ] Baseline tuned **as much as** your own method. Never handicap it.
|
||||
- [ ] All else held equal (e.g. comparable parameter counts across architecture variants).
|
||||
- [ ] At least 3 seeds, ideally 10 or more; plot mean and std dev.
|
||||
- [ ] Final runs launched fresh and **precommitted** — not the best or most interesting runs.
|
||||
- [ ] One ablation per design decision, so each claim is separately supported.
|
||||
- [ ] Checked the idea has not already been published.
|
||||
|
||||
## Tells and smells
|
||||
|
||||
- **A learning curve with no seed band** is telling you less than it looks like.
|
||||
- **`AverageEpRet` on an off-policy algorithm** is the noisy behavior policy, not performance.
|
||||
Use the plotter's `Performance` alias.
|
||||
- **Evaluating SAC without `--deterministic`** measures the wrong policy.
|
||||
- **A model-based agent that scores well but behaves badly** is exploiting model bias — the
|
||||
same shape as a policy exploiting Q-function errors.
|
||||
- **"Fits the Bellman equation well"** carries no guarantee of good policy performance.
|
||||
- **A single missing `squeeze`** in a critic is the archetype of silent failure.
|
||||
- **A paper's full trick list** is usually more than strictly necessary — try simplifications.
|
||||
- **A library abstraction** is good for reuse across algorithms and a cost when learning one.
|
||||
|
||||
## Quick commands
|
||||
|
||||
```bash
|
||||
python -m spinup.run ppo --env Walker2d-v2 --exp_name walker --seed 0 10 20
|
||||
python -m spinup.run test_policy data/walker/walker_s0 # add -d for SAC
|
||||
python -m spinup.run plot data/walker # prefix autocompletes
|
||||
python -m spinup.run [algo] --help # every kwarg is a flag
|
||||
```
|
||||
|
|
@ -0,0 +1,56 @@
|
|||
# Glossary
|
||||
|
||||
**Action space** — the set of all valid actions in an environment; discrete or continuous (Ch 7)
|
||||
**Action-value function** — see Q-function (Ch 7)
|
||||
**Advantage function** — `A^pi(s,a) = Q^pi(s,a) - V^pi(s)`; how much better a specific action is than randomly selecting one per pi (Ch 7, Ch 9)
|
||||
**Agent** — the learner and decider; often used interchangeably with "policy" (Ch 7)
|
||||
**Backtracking line search** — shrink the step by `alpha^j` until the exact KL constraint holds and surrogate advantage is positive (Ch 15)
|
||||
**Baseline** — any state-only function `b(s)` subtractable from the policy gradient weight without changing its expectation (Ch 9)
|
||||
**Bellman backup** — the right-hand side of a Bellman equation: reward plus next value (Ch 7)
|
||||
**Bellman equation** — self-consistency condition: the value of your starting point is the reward you expect from being there plus the value of where you land next (Ch 7)
|
||||
**Categorical policy** — stochastic policy for discrete actions, built like a classifier: logits then softmax (Ch 7)
|
||||
**Clipped double-Q learning** — learn two Q-functions and use the smaller value to form the target (Ch 18, Ch 19)
|
||||
**Deadly triad** — function approximation + bootstrapping + off-policy data, jointly causing instability in value learning (Ch 8)
|
||||
**Diagonal Gaussian policy** — stochastic policy for continuous actions; mean network plus a diagonal covariance represented as a vector of log stds (Ch 7)
|
||||
**Discount factor (gamma)** — in (0,1); makes an infinite reward sum converge and encodes "cash now beats cash later" (Ch 7)
|
||||
**EGLP lemma** — Expected Grad-Log-Prob: `E_{x~P_theta}[grad log P_theta(x)] = 0` (Ch 9)
|
||||
**Entropy** — `H(P) = E_{x~P}[-log P(x)]`; how random a random variable is (Ch 19)
|
||||
**Entropy regularization** — adding an entropy bonus to the reward at each timestep, scaled by `alpha` (Ch 19)
|
||||
**Epoch** — one experience-collection phase plus one policy update (Ch 9)
|
||||
**ExperimentGrid** — Spinning Up's hyperparameter ablation tool; backs the CLI (Ch 4, Ch 20)
|
||||
**Fully / partially observed** — whether the agent sees the complete state or only part of it (Ch 7)
|
||||
**GAE** — Generalized Advantage Estimation; used by Spinning Up's VPG, TRPO and PPO (Ch 9)
|
||||
**Log-derivative trick** — `grad P = P * grad log P` (Ch 9)
|
||||
**Markov property** — transitions depend only on the most recent state and action (Ch 7)
|
||||
**MDP** — Markov Decision Process, the 5-tuple `<S, A, R, P, rho_0>` (Ch 7)
|
||||
**Model** — a function predicting state transitions and rewards (Ch 8)
|
||||
**Model-based / model-free** — whether the agent has or learns a model (Ch 8)
|
||||
**MPC** — model-predictive control: plan over a window, execute the first action, discard the rest, re-plan (Ch 8)
|
||||
**MSBE** — Mean-Squared Bellman Error, the loss deep Q-learning minimizes (Ch 17)
|
||||
**Natural Policy Gradient** — TRPO's analytic solution before the line search is added (Ch 15)
|
||||
**Observation** — a partial description of a state, possibly omitting information (Ch 7)
|
||||
**Off-policy** — updates may use data collected at any point in training (Ch 3, Ch 8)
|
||||
**On-policy** — updates use only data from the most recent policy (Ch 3, Ch 8)
|
||||
**Optimal policy (pi\*)** — `argmax_pi J(pi)` (Ch 7)
|
||||
**Parameterized policy** — a policy whose output depends on adjustable parameters, e.g. network weights (Ch 7)
|
||||
**Performance** — the plotter's family-aware alias, resolving to `AverageEpRet` (on-policy) or `AverageTestEpRet` (off-policy) (Ch 6)
|
||||
**Policy** — the rule for choosing actions; deterministic `mu(s)` or stochastic `pi(.|s)` (Ch 7)
|
||||
**Policy delay** — TD3's ratio of Q-updates to policy updates; recommended 2 (Ch 18)
|
||||
**Policy gradient** — `grad_theta J(pi_theta)` (Ch 9)
|
||||
**Polyak averaging** — target update `phi_targ <- rho*phi_targ + (1-rho)*phi`; `polyak` in the code (Ch 17)
|
||||
**Q-function** — `Q^pi(s,a)`, expected return from taking a in s then following pi (Ch 7)
|
||||
**Replay buffer** — the stored set of past transitions that off-policy algorithms sample from (Ch 17)
|
||||
**Reparameterization trick** — sample by a deterministic function of state, parameters and independent noise (Ch 19)
|
||||
**Return** — cumulative reward over a trajectory; finite-horizon undiscounted or infinite-horizon discounted (Ch 7)
|
||||
**Reward-to-go** — `Rhat_t = sum_{t'=t}^T R(s_t', a_t', s_{t'+1})` (Ch 9)
|
||||
**Silent failure** — broken RL code that runs correctly and simply never learns (Ch 10, Ch 12)
|
||||
**Squashed Gaussian policy** — a Gaussian passed through `tanh` to bound actions; SAC's policy (Ch 19)
|
||||
**start_steps** — initial phase of uniform-random actions used by DDPG, TD3 and SAC (Ch 17)
|
||||
**State** — a complete description of the world, with nothing hidden (Ch 7)
|
||||
**Surrogate advantage** — importance-weighted advantage measuring new-policy performance on old-policy data (Ch 15)
|
||||
**Surrogate objective** — a local approximation of `J(pi_theta)` that is safe to maximize (Ch 8, Ch 16)
|
||||
**Target network** — a time-lagged copy of a network used to compute stable Bellman targets (Ch 17)
|
||||
**Target policy smoothing** — clipped noise added to the target action to prevent exploiting sharp Q peaks (Ch 18)
|
||||
**Trajectory (tau)** — a sequence of states and actions; also called episode or rollout (Ch 7)
|
||||
**Trust region** — a KL-divergence bound on how far the new policy may move from the old (Ch 15)
|
||||
**Value function** — `V^pi(s)`, expected return from s following pi (Ch 7)
|
||||
|
|
@ -0,0 +1,131 @@
|
|||
# Patterns
|
||||
|
||||
Concrete techniques from Spinning Up, each with when to use it, how it works, and what it costs.
|
||||
|
||||
## Reward-to-Go Weighting
|
||||
**When to use** — always, over full-trajectory-return weighting. It is a two-line change.
|
||||
**How** — weight each `grad log pi(a_t|s_t)` by `sum_{t'=t}^T r_t'` instead of `R(tau)`.
|
||||
**Trade-offs** — strictly reduces variance at no cost. The dropped terms had zero mean and
|
||||
nonzero variance, so removing them removes pure noise and cuts the sample trajectories needed.
|
||||
(Ch 9)
|
||||
|
||||
## Value Function Baseline
|
||||
**When to use** — any policy gradient method. Standard in VPG, TRPO, PPO and A2C.
|
||||
**How** — subtract `V_phi(s_t)` from the weight; fit `V_phi` by minimizing
|
||||
`E[(V_phi(s_t) - Rhat_t)^2]` with one or more gradient steps per epoch, starting from the
|
||||
previous parameters, so it tracks the most recent policy.
|
||||
**Trade-offs** — provably unbiased by EGLP; reduces variance and gives faster, more stable
|
||||
learning. Costs a second network and its fitting budget. Under-fitting it is one of the most
|
||||
drastic performance failures in policy gradients (Exercise 2.1). (Ch 9, Ch 12)
|
||||
|
||||
## Trust Region via KL Constraint
|
||||
**When to use** — when a single bad policy gradient step can collapse performance, and you
|
||||
can afford second-order machinery.
|
||||
**How** — maximize surrogate advantage subject to `Dbar_KL <= delta`; Taylor-expand, solve
|
||||
analytically via Lagrangian duality, then backtracking line search until the exact KL
|
||||
constraint holds and surrogate advantage is positive. Use conjugate gradient for `H^{-1}g`.
|
||||
**Trade-offs** — buys monotone-ish, fast improvement and larger safe steps. Costs complexity,
|
||||
Hessian-vector products, and (in Spinning Up) no PyTorch implementation. (Ch 15)
|
||||
|
||||
## Clipped Surrogate Objective
|
||||
**When to use** — as the first-order replacement for a trust region. The default modern choice.
|
||||
**How** — `L = min(ratio * A, g(eps, A))` where `g` caps at `(1+eps)A` for positive advantage
|
||||
and `(1-eps)A` for negative. Take multiple minibatch SGD steps per batch.
|
||||
**Trade-offs** — much simpler than TRPO and empirically at least as good, but it removes the
|
||||
*incentive* to move far rather than guaranteeing you do not. Pair it with KL early stopping.
|
||||
(Ch 16)
|
||||
|
||||
## KL Early Stopping
|
||||
**When to use** — alongside clipping, whenever you take many gradient steps per batch.
|
||||
**How** — if the mean KL-divergence of the new policy from the old exceeds a threshold, stop
|
||||
taking gradient steps for this batch.
|
||||
**Trade-offs** — one extra measurement per step; catches the drift the clip alone allows.
|
||||
Other PPO implementations use different tricks here. (Ch 16)
|
||||
|
||||
## Experience Replay Buffer
|
||||
**When to use** — every off-policy value-learning algorithm.
|
||||
**How** — store `(s,a,r,s',d)` tuples; sample minibatches uniformly for MSBE minimization.
|
||||
Licensed by the fact that the Bellman equation is indifferent to how transitions were collected.
|
||||
**Trade-offs** — wrong in both directions. Too small and you overfit to the most recent data
|
||||
and things break; too large and learning slows. It needs tuning. (Ch 17)
|
||||
|
||||
## Target Networks
|
||||
**When to use** — any MSBE minimization, because the target otherwise depends on the
|
||||
parameters being trained.
|
||||
**How** — keep a lagged copy. DQN-style: copy every fixed number of steps. DDPG-style:
|
||||
polyak average once per main update, `phi_targ <- rho*phi_targ + (1-rho)*phi`, `rho` near 1.
|
||||
**Trade-offs** — stabilizes learning; adds a network and a hyperparameter, and slows the
|
||||
propagation of new information into the target. (Ch 17)
|
||||
|
||||
## Amortizing a Continuous Argmax into a Policy
|
||||
**When to use** — an inner `max` over a continuous variable inside a loop you run constantly.
|
||||
**How** — assume differentiability with respect to that variable, learn `mu_theta(s)` by
|
||||
gradient ascent on `Q_phi(s, mu_theta(s))`, and substitute `max_a Q(s,a) ~= Q(s, mu(s))`.
|
||||
**Trade-offs** — turns an intractable per-step optimization into a forward pass. Costs an
|
||||
approximation whose errors the policy will actively exploit. (Ch 17)
|
||||
|
||||
## Clipped Double-Q
|
||||
**When to use** — whenever a learned Q-function is being maximized over by a policy, i.e.
|
||||
the whole DDPG family.
|
||||
**How** — learn two Q-functions, use `min` of the two target values as the shared regression
|
||||
target for both.
|
||||
**Trade-offs** — fends off overestimation, which is DDPG's dominant failure. Costs a second
|
||||
critic and introduces some underestimation bias. TD3 uses `Q_phi_1` alone in the policy loss;
|
||||
SAC uses the `min` there too. (Ch 18, Ch 19)
|
||||
|
||||
## Delayed Policy Updates
|
||||
**When to use** — when policy updates destabilize the Bellman target.
|
||||
**How** — update the policy and all target networks once per `policy_delay` critic updates;
|
||||
the paper recommends 2.
|
||||
**Trade-offs** — damps volatility; slows policy improvement per environment step. (Ch 18)
|
||||
|
||||
## Target Policy Smoothing
|
||||
**When to use** — deterministic policies over continuous actions.
|
||||
**How** — `a'(s') = clip(mu_targ(s') + clip(eps, -c, c), a_Low, a_High)`, `eps ~ N(0, sigma)`.
|
||||
**Trade-offs** — regularizes Q along the action dimension so a spurious sharp peak cannot be
|
||||
exploited. A stochastic policy (SAC) gets a similar effect for free. (Ch 18, Ch 19)
|
||||
|
||||
## Entropy Regularization
|
||||
**When to use** — when premature convergence to a bad local optimum is the risk, or when you
|
||||
want an explicit explore-exploit dial.
|
||||
**How** — add `alpha * H(pi(.|s_t))` to the reward at each timestep; propagate the term into
|
||||
the value functions and the Bellman target.
|
||||
**Trade-offs** — higher `alpha` means more exploration and faster later learning; the right
|
||||
value is environment-specific and needs careful tuning. The entropy-constrained variant that
|
||||
adapts `alpha` is generally preferred by practitioners over the fixed one. (Ch 19)
|
||||
|
||||
## Reparameterization Trick for Stochastic Policies
|
||||
**When to use** — differentiating through an expectation whose distribution depends on the
|
||||
parameters.
|
||||
**How** — sample by a deterministic function of state, parameters and independent noise:
|
||||
`a~ = tanh(mu_theta(s) + sigma_theta(s) * xi)`, `xi ~ N(0,I)`, converting the expectation over
|
||||
actions into an expectation over noise.
|
||||
**Trade-offs** — makes the policy differentiable end to end. The `tanh` squash bounds actions
|
||||
but changes the distribution; log-probabilities are still closed-form. (Ch 19)
|
||||
|
||||
## Uniform Random Warm-Up (start_steps)
|
||||
**When to use** — every off-policy algorithm here, at the beginning of training.
|
||||
**How** — for a fixed number of steps take actions sampled uniformly over valid actions, then
|
||||
switch to normal exploration.
|
||||
**Trade-offs** — fills the replay buffer with diverse data before the policy can bias it.
|
||||
Wasted steps if the environment is expensive. (Ch 17, Ch 18, Ch 19)
|
||||
|
||||
## Multi-Seed Reporting with Variance Bands
|
||||
**When to use** — every reported RL result.
|
||||
**How** — at least 3 seeds, 10 or more to be thorough; plot the mean as a solid line and the
|
||||
standard deviation as a shaded band; smooth for display only.
|
||||
**Trade-offs** — multiplies compute by the seed count. Without it, deep RL's seed sensitivity
|
||||
means two seed groups can look like different distributions entirely. (Ch 10, Ch 13)
|
||||
|
||||
## Precommitted Final Runs
|
||||
**When to use** — before reporting any comparison.
|
||||
**How** — use the tuning stage to form hypotheses, then launch fresh final experiments for
|
||||
every method compared and commit in advance to reporting whatever comes out.
|
||||
**Trade-offs** — a weak form of preregistration; costs one more full experiment round and
|
||||
removes the largest source of accidental self-deception. (Ch 10)
|
||||
|
||||
## Per-Claim Ablation
|
||||
**When to use** — any method with more than one design decision, which is all of them.
|
||||
**How** — swap out or remove each design element separately and measure.
|
||||
**Trade-offs** — turns one bundled claim into several separately-confident ones. Costs one
|
||||
experiment per element. Also the fastest way to learn which parts you can delete. (Ch 10)
|
||||
Loading…
Add table
Reference in a new issue