diff --git a/.claude-plugin/marketplace.json b/.claude-plugin/marketplace.json index 687c91b4..6aca8a8f 100644 --- a/.claude-plugin/marketplace.json +++ b/.claude-plugin/marketplace.json @@ -8,7 +8,7 @@ "homepage": "https://github.com/alirezarezvani/claude-skills", "repository": "https://github.com/alirezarezvani/claude-skills", "metadata": { - "description": "386 production-ready skills across 20 domains (engineering, engineering-core, marketing, product, c-level, c-level-agents, compliance-os, project management, RA/QM, business growth, finance, productivity, marketing top-level, research, research-ops, business-operations, commercial, markdown-html, loop-library, plus standards). 723 Python tools, 838 reference guides, 116 agents (cs-* + personas), 146 slash commands across 97 marketplace plugins. v2.11.2 vendors engineering/skillopt-sleep — a verbatim copy of microsoft/SkillOpt's stdlib-only skillopt_sleep engine + Claude Code plugin surface, giving a local agent a nightly gated self-improvement cycle (read-only session harvest -> mine -> offline replay -> held-out-gated CLAUDE.md/SKILL.md edits -> staged for explicit /skillopt-sleep adopt). productivity/fable-goal (unreleased, post-v2.11.1) converts a rambling description of a desired outcome into one polished /goal prompt for a fresh autonomous session. v2.11.1 turns product-team and project-management into agent-harness domains: fork-orchestrators with deterministic goal routers, a Jira MCP snapshot bridge (Kanban flow metrics + Monte Carlo forecasting), a delegation-governance loop gate, a continuous-discovery cadence tracker, and an Opportunity Solution Tree linter, with /cs:pm and /cs:product command families. v2.10.3 completes the markdown-html domain with md-slides — slide-deck converter (arrow-key / Space / PgDn / Home/End / P keyboard navigation + presenter mode with split-view clock + speaker notes + next-slide preview + URL-hash deep linking like #3 for direct slide jumps + @media print page-per-slide for browser-native PDF export). Reuses md-document's markdown parser; vanilla JS only (no framework runtime); Prism.js opt-in via --syntax. Joins md-review (v2.10.2 code-review converter), md-document (v2.10.1 long-form converter), and the v2.10.0 foundation (orchestrator + design-system). Compatible with Claude Code, Codex CLI, Gemini CLI, Cursor, OpenClaw, Hermes Agent, Mistral Vibe, and 5 more coding agents.", + "description": "387 production-ready skills across 20 domains (engineering, engineering-core, marketing, product, c-level, c-level-agents, compliance-os, project management, RA/QM, business growth, finance, productivity, marketing top-level, research, research-ops, business-operations, commercial, markdown-html, loop-library, plus standards). 723 Python tools, 838 reference guides, 117 agents (cs-* + personas), 147 slash commands across 98 marketplace plugins. v2.11.2 vendors engineering/skillopt-sleep — a verbatim copy of microsoft/SkillOpt's stdlib-only skillopt_sleep engine + Claude Code plugin surface, giving a local agent a nightly gated self-improvement cycle (read-only session harvest -> mine -> offline replay -> held-out-gated CLAUDE.md/SKILL.md edits -> staged for explicit /skillopt-sleep adopt). productivity/fable-goal (unreleased, post-v2.11.1) converts a rambling description of a desired outcome into one polished /goal prompt for a fresh autonomous session. v2.11.1 turns product-team and project-management into agent-harness domains: fork-orchestrators with deterministic goal routers, a Jira MCP snapshot bridge (Kanban flow metrics + Monte Carlo forecasting), a delegation-governance loop gate, a continuous-discovery cadence tracker, and an Opportunity Solution Tree linter, with /cs:pm and /cs:product command families. v2.10.3 completes the markdown-html domain with md-slides — slide-deck converter (arrow-key / Space / PgDn / Home/End / P keyboard navigation + presenter mode with split-view clock + speaker notes + next-slide preview + URL-hash deep linking like #3 for direct slide jumps + @media print page-per-slide for browser-native PDF export). Reuses md-document's markdown parser; vanilla JS only (no framework runtime); Prism.js opt-in via --syntax. Joins md-review (v2.10.2 code-review converter), md-document (v2.10.1 long-form converter), and the v2.10.0 foundation (orchestrator + design-system). Compatible with Claude Code, Codex CLI, Gemini CLI, Cursor, OpenClaw, Hermes Agent, Mistral Vibe, and 5 more coding agents.", "version": "2.12.0" }, "plugins": [ @@ -2050,6 +2050,28 @@ "engineering" ], "category": "development" + }, + { + "name": "spinning-up-deep-rl", + "source": "./engineering/spinning-up-deep-rl", + "description": "Knowledge base compiled from OpenAI's Spinning Up in Deep RL (MIT, Joshua Achiam) by engineering/book-to-skill. A resident core carries the RL optimization problem, the model-free taxonomy, the policy-gradient template with its five valid weights, the safe-step family (VPG to TRPO to PPO) and the overestimation family (DDPG to TD3 and SAC); 20 on-demand chapters cover key concepts and MDPs, the algorithm taxonomy and model bias, the policy gradient derivation with the log-derivative trick and EGLP lemma, Achiam's researcher essay, the key-papers topic map, the exercises including the silent DDPG broadcasting bug, the benchmark parity disclosure, one chapter per algorithm, and the logger/MPI/ExperimentGrid utilities. Ships a glossary, a patterns file with 16 techniques, a decision cheatsheet with thresholds, a cs-spinning-up-deep-rl agent and /cs:spinning-up-deep-rl. Structured study notes, not a reproduction of the source.", + "version": "1.0.0", + "author": { + "name": "Alireza Rezvani" + }, + "keywords": [ + "knowledge-base", + "book-to-skill", + "reinforcement-learning", + "deep-rl", + "policy-gradient", + "ppo", + "sac", + "td3", + "openai-spinning-up", + "engineering" + ], + "category": "development" } ] } diff --git a/CHANGELOG.md b/CHANGELOG.md index 886cabf6..8f52b857 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,46 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +### Added — engineering/spinning-up-deep-rl: the first book compiled by book-to-skill + +Knowledge-base plugin compiled end-to-end by `engineering/book-to-skill` from OpenAI's +[Spinning Up in Deep RL](https://spinningup.openai.com/) (MIT, Copyright (c) 2018 OpenAI; +primarily developed by Joshua Achiam). 20 chapters, a glossary, a patterns file and a +decision cheatsheet, behind a 2,101-token resident core. + +- **The full pipeline, not a hand-write.** `openai/spinningup` cloned, its `docs/` + reStructuredText tree (38 files, ~37k words, ~49K tokens) run through + `extract_document.py --mode technical` → analysis → chapter files → supporting files → + master `SKILL.md` → `book_skill_validator.py` → `skill_plugin_emitter.py`. The validator + passes clean in `--strict` mode and every file is inside budget. +- **Rights basis `open-license`, stated and honoured.** The emitter's Step-11 gate refuses a + shareable package without one. MIT permits derivative distribution; upstream's notice is + reproduced in full in the plugin's `LICENSE` beside this package's own, and `README.md` + names the source, the author and the source's frozen version. +- **Structure follows the source's own `toctree`.** User documentation (ch01-06), Introduction + to RL Parts 1-3 (ch07-09), resources — the researcher essay, key papers, exercises, + benchmarks (ch10-13), one chapter per algorithm in lineage order (ch14-19: VPG → TRPO → PPO, + DDPG → TD3 → SAC), and the logger / MPI / ExperimentGrid utilities (ch20). +- **The cheatsheet carries the judgment a glossary cannot** — the under-5-minute debug + turnaround, the 3-seed minimum (10+ to be thorough), family-specific benchmark network + defaults, and Spinning Up's own parity disclosure: DDPG/TD3/SAC are research-grade, + VPG/TRPO/PPO are not, and the docs say to use OpenAI Baselines for those. +- **Counters:** skills 386 → 387; agents 116 → 117; commands 146 → 147; plugins 97 → 98. + Tools and references unchanged — a compiled knowledge base ships notes, not scripts. + +### Fixed — book-to-skill's plugin emitter produced manifests this repo's CI rejects + +`skill_plugin_emitter.py` wrote its whole `source` provenance block into `plugin.json`, with an +inline comment asserting that `source` and `attribution` were approved extension fields. That had +been true and no longer was: Claude Code rejects an entire manifest on any unrecognized key +(issue #954), and `scripts/check_plugin_json.py` hard-fails such a manifest, pointing at +`.claude-plugin/authoring-notes.json` instead. Every package the emitter produced therefore failed +the blocking CI gate the moment it was committed — a defect at the very last step of the pipeline, +which is why it had gone unnoticed. `_plugin_manifest()` now emits spec fields only and a new +`_authoring_notes()` writes the sidecar. Recorded as deviation 26 in +`engineering/book-to-skill/README.md`. The printed `marketplace.json` snippet is unchanged: `source` +is a valid key there, which is how it leaked into the manifest originally. + ### Added — marketing/linkedin: organic LinkedIn presence with the platform rules in code New `marketing/linkedin/` plugin, answering diff --git a/CLAUDE.md b/CLAUDE.md index 89d1522e..19819b6f 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -6,7 +6,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co This is a **comprehensive skills library** for Claude AI and Claude Code - reusable, production-ready skill packages that bundle domain expertise, best practices, analysis tools, and strategic frameworks. The repository provides modular skills that teams can download and use directly in their workflows. -**Current Scope:** 386 production-ready skills across 20 domains with 723 Python automation tools, 838 reference guides, 116 agents (cs-* + 7 personas), and 146 slash commands, distributed as 97 marketplace plugins. Headline counters are derived from the tree by `scripts/derive_counters.py` (run with `--check` to verify the docs still match). **v2.11.2 (current)** vendors **engineering/skillopt-sleep/** — started as a verbatim, byte-for-byte copy of `microsoft/SkillOpt`'s `skillopt_sleep` engine (stdlib-only, zero third-party deps) and its Claude Code plugin surface (`skills/`, `hooks/`, `commands/`, `scripts/`), then received 23 targeted patches after ten rounds of adversarial review (see `engineering/skillopt-sleep/README.md`'s numbered "Deviations from upstream" list, the authoritative source — re-apply all 23 on re-vendor). Gives a local agent a nightly "sleep cycle": read-only harvest of past Claude Code session transcripts → mine recurring tasks → replay offline on the user's own API budget → consolidate into `CLAUDE.md`/`SKILL.md` edits behind a held-out validation gate → stage for review; nothing live changes until an explicit `/skillopt-sleep adopt` (which backs up first). Default `mock` backend spends no API budget. The heavier `skillopt` *training* package (benchmark-driven, needs `numpy`/`openai`/`azure-*` + hand-labeled train/val/test data per task) was deliberately **not** vendored — it optimizes one narrow, scoreable task at a time, which doesn't fit this repo's broad domain-expertise skills or its no-ML-in-scripts/no-test-framework conventions; `skillopt_sleep` mines its "benchmark" from real usage instead, which does fit. Attribution preserved in `.claude-plugin/authoring-notes.json` + `LICENSE` + `README.md` (MIT, © Microsoft Corporation / Yifan Yang), following the same verbatim-vendor pattern as `loop-library/`. **Unreleased (post-v2.11.2, PR #961 merged)** adds the **agent-launcher/** top-level domain — a plugin re-implementation of Anthropic's `launch-your-agent` reference skill (Apache-2.0; independent, not a fork) for building **Claude Managed Agents (CMA)** in the user's own account. Every session starts with a goal (`./my-agent/goal.json`, surfaced by an opt-in `AGENT_LAUNCHER_SESSION=1` SessionStart hook + `/cs:goal`); `loop_compiler.py` compiles that goal into a **bounded grade→iterate loop** (CMA `user.define_outcome` self-grading, `max_iterations` 1..20), a **recurring POSIX-cron scheduled-deployment loop**, or a **single-pass interview→stage→launch workflow**. 6 skills (orchestrator `context: fork` + interview + stage-launch + grade-iterate + run-without-you + wrap-up), 18 stdlib-only deterministic scaffolder tools (NO network/API calls — live launches emitted as BYOK curl that never prints the key), 4 agents, 8 `/cs:*` commands, opt-in hooks, 5 shared references, 4 assets; validators enforce CMA limits (≤20 skills/session, ≤8 memory stores, depth-1 multiagent, `max_iterations` ≤20, ≤1000 deployments/org). Distinct from `engineering/agent-harness` (generic bounded loop over any domain) and `engineering/write-a-skill` (authors Claude Code skills, not CMAs). **Unreleased (post-v2.11.2)** ships the **productivity coverage expansion** — public audit record `audit/productivity-2026-07/` (all 7 legacy skills scored, 24/24 scripts smoke-tested, coverage map vs the personal-productivity canon) + 3 gap-filling plugins, each with a cs-* agent, /cs:* commands, 3 stdlib scripts and 3 cited references: **weekly-review** (GTD loop; review-gate refuses COMPLETE while a mandatory GET CURRENT step is missing), **deep-work** (time-block planner refusing >4h deep demand, shallow-work budget auditor, focus-session logger), **meetings** (MEET/ASYNC/NOT-READY cost gate, outcome-required agenda builder, action-item extractor with ORPHAN/NO-DUE flags). **Unreleased (post-v2.11.1)** added **productivity/fable-goal** — converts a rambling description of a desired outcome into one polished, copy-paste `/goal` prompt for a fresh autonomous session (ported from `duncan-buildroom/freeskills`). **v2.11.1 (complete)** upgrades **product-team/** and **project-management/** into agent-harness domains: both prose routers rebuilt as `context: fork` orchestrators with deterministic goal routers (exit-code route/ask/refuse), a Jira MCP snapshot bridge (Kanban-Guide-2025 flow metrics + seeded Monte Carlo forecasts, verified end-to-end into velocity_analyzer), a delegation-governance loop gate (human owner / reviewer / machine-checkable acceptance / close refusal), a Torres continuous-discovery cadence tracker + Opportunity Solution Tree linter, cs-pm-orchestrator + cs-product-orchestrator agents, and /cs:pm|grill-pm|pm-loop + /cs:product|grill-product|product-loop commands — plus the public audit record `audit/pm-product-agentic-2026-07/` (AR-rubric scores for all 26 skills, research-backed improvement fields, executable verification criteria). **v2.9.0 (complete)** added the **research-ops/** top-level domain — enterprise Research Operations (orchestrator + clinical-research + research-finance + market-research + product-research), the managed counterpart to the academic research/ domain, with `context: fork` orchestration and a Matt Pocock "Forcing-question library" in every SKILL.md plus `/cs:grill-research-ops`. **v2.8.0 (complete)** added 2 new top-level domains — **business-operations/** (7 internal-ops skills: orchestrator + process-mapper + vendor-management + capacity-planner + internal-comms + knowledge-ops + procurement-optimizer) and **commercial/** (8 per-deal-economics skills: orchestrator + pricing-strategist + deal-desk + partnerships-architect + channel-economics + commercial-policy + rfp-responder + commercial-forecaster) — with orchestrator skills using `context: fork` for chaining, Matt Pocock docs-anchored "Forcing-question library" in every SKILL.md, plus `/cs:grill-bizops` and `/cs:grill-commercial`. **v2.8.2** adds a productivity-shaped `handoff` skill (sibling to engineering/handoff) inspired by Matt Pocock — first-run setup with configurable save location, redaction linter, SessionStart + SessionEnd hooks, fidelity self-check, `--refresh` flag. **v2.8.1** upgraded the engineering role-skills (senior-fullstack / senior-frontend / senior-backend) with karpathy-coder + Matt Pocock decision engines + per-role forcing questions. v2.7.3 ports `alirezarezvani/aeo-box` — AEO (Answer Engine Optimization) skill into marketing-skill/ + security-guidance PreToolUse hook into engineering/. v2.7.0 added 13 Path-B skills across 3 top-level domains (productivity, marketing, research). v2.6.0 added 4 Matt Pocock-derived productivity skills. +**Current Scope:** 387 production-ready skills across 20 domains with 723 Python automation tools, 838 reference guides, 117 agents (cs-* + 7 personas), and 147 slash commands, distributed as 98 marketplace plugins. Headline counters are derived from the tree by `scripts/derive_counters.py` (run with `--check` to verify the docs still match). **v2.11.2 (current)** vendors **engineering/skillopt-sleep/** — started as a verbatim, byte-for-byte copy of `microsoft/SkillOpt`'s `skillopt_sleep` engine (stdlib-only, zero third-party deps) and its Claude Code plugin surface (`skills/`, `hooks/`, `commands/`, `scripts/`), then received 23 targeted patches after ten rounds of adversarial review (see `engineering/skillopt-sleep/README.md`'s numbered "Deviations from upstream" list, the authoritative source — re-apply all 23 on re-vendor). Gives a local agent a nightly "sleep cycle": read-only harvest of past Claude Code session transcripts → mine recurring tasks → replay offline on the user's own API budget → consolidate into `CLAUDE.md`/`SKILL.md` edits behind a held-out validation gate → stage for review; nothing live changes until an explicit `/skillopt-sleep adopt` (which backs up first). Default `mock` backend spends no API budget. The heavier `skillopt` *training* package (benchmark-driven, needs `numpy`/`openai`/`azure-*` + hand-labeled train/val/test data per task) was deliberately **not** vendored — it optimizes one narrow, scoreable task at a time, which doesn't fit this repo's broad domain-expertise skills or its no-ML-in-scripts/no-test-framework conventions; `skillopt_sleep` mines its "benchmark" from real usage instead, which does fit. Attribution preserved in `.claude-plugin/authoring-notes.json` + `LICENSE` + `README.md` (MIT, © Microsoft Corporation / Yifan Yang), following the same verbatim-vendor pattern as `loop-library/`. **Unreleased (post-v2.11.2, PR #961 merged)** adds the **agent-launcher/** top-level domain — a plugin re-implementation of Anthropic's `launch-your-agent` reference skill (Apache-2.0; independent, not a fork) for building **Claude Managed Agents (CMA)** in the user's own account. Every session starts with a goal (`./my-agent/goal.json`, surfaced by an opt-in `AGENT_LAUNCHER_SESSION=1` SessionStart hook + `/cs:goal`); `loop_compiler.py` compiles that goal into a **bounded grade→iterate loop** (CMA `user.define_outcome` self-grading, `max_iterations` 1..20), a **recurring POSIX-cron scheduled-deployment loop**, or a **single-pass interview→stage→launch workflow**. 6 skills (orchestrator `context: fork` + interview + stage-launch + grade-iterate + run-without-you + wrap-up), 18 stdlib-only deterministic scaffolder tools (NO network/API calls — live launches emitted as BYOK curl that never prints the key), 4 agents, 8 `/cs:*` commands, opt-in hooks, 5 shared references, 4 assets; validators enforce CMA limits (≤20 skills/session, ≤8 memory stores, depth-1 multiagent, `max_iterations` ≤20, ≤1000 deployments/org). Distinct from `engineering/agent-harness` (generic bounded loop over any domain) and `engineering/write-a-skill` (authors Claude Code skills, not CMAs). **Unreleased (post-v2.11.2)** ships the **productivity coverage expansion** — public audit record `audit/productivity-2026-07/` (all 7 legacy skills scored, 24/24 scripts smoke-tested, coverage map vs the personal-productivity canon) + 3 gap-filling plugins, each with a cs-* agent, /cs:* commands, 3 stdlib scripts and 3 cited references: **weekly-review** (GTD loop; review-gate refuses COMPLETE while a mandatory GET CURRENT step is missing), **deep-work** (time-block planner refusing >4h deep demand, shallow-work budget auditor, focus-session logger), **meetings** (MEET/ASYNC/NOT-READY cost gate, outcome-required agenda builder, action-item extractor with ORPHAN/NO-DUE flags). **Unreleased (post-v2.11.1)** added **productivity/fable-goal** — converts a rambling description of a desired outcome into one polished, copy-paste `/goal` prompt for a fresh autonomous session (ported from `duncan-buildroom/freeskills`). **v2.11.1 (complete)** upgrades **product-team/** and **project-management/** into agent-harness domains: both prose routers rebuilt as `context: fork` orchestrators with deterministic goal routers (exit-code route/ask/refuse), a Jira MCP snapshot bridge (Kanban-Guide-2025 flow metrics + seeded Monte Carlo forecasts, verified end-to-end into velocity_analyzer), a delegation-governance loop gate (human owner / reviewer / machine-checkable acceptance / close refusal), a Torres continuous-discovery cadence tracker + Opportunity Solution Tree linter, cs-pm-orchestrator + cs-product-orchestrator agents, and /cs:pm|grill-pm|pm-loop + /cs:product|grill-product|product-loop commands — plus the public audit record `audit/pm-product-agentic-2026-07/` (AR-rubric scores for all 26 skills, research-backed improvement fields, executable verification criteria). **v2.9.0 (complete)** added the **research-ops/** top-level domain — enterprise Research Operations (orchestrator + clinical-research + research-finance + market-research + product-research), the managed counterpart to the academic research/ domain, with `context: fork` orchestration and a Matt Pocock "Forcing-question library" in every SKILL.md plus `/cs:grill-research-ops`. **v2.8.0 (complete)** added 2 new top-level domains — **business-operations/** (7 internal-ops skills: orchestrator + process-mapper + vendor-management + capacity-planner + internal-comms + knowledge-ops + procurement-optimizer) and **commercial/** (8 per-deal-economics skills: orchestrator + pricing-strategist + deal-desk + partnerships-architect + channel-economics + commercial-policy + rfp-responder + commercial-forecaster) — with orchestrator skills using `context: fork` for chaining, Matt Pocock docs-anchored "Forcing-question library" in every SKILL.md, plus `/cs:grill-bizops` and `/cs:grill-commercial`. **v2.8.2** adds a productivity-shaped `handoff` skill (sibling to engineering/handoff) inspired by Matt Pocock — first-run setup with configurable save location, redaction linter, SessionStart + SessionEnd hooks, fidelity self-check, `--refresh` flag. **v2.8.1** upgraded the engineering role-skills (senior-fullstack / senior-frontend / senior-backend) with karpathy-coder + Matt Pocock decision engines + per-role forcing questions. v2.7.3 ports `alirezarezvani/aeo-box` — AEO (Answer Engine Optimization) skill into marketing-skill/ + security-guidance PreToolUse hook into engineering/. v2.7.0 added 13 Path-B skills across 3 top-level domains (productivity, marketing, research). v2.6.0 added 4 Matt Pocock-derived productivity skills. **Key Distinction**: This is NOT a traditional application. It's a library of skill packages meant to be extracted and deployed by users into their own Claude workflows. @@ -190,6 +190,50 @@ New `marketing/linkedin/` plugin answering [discussion #934](https://github.com/ --- +**Unreleased (post-v2.12.0) — engineering/spinning-up-deep-rl (the first book compiled by book-to-skill):** + +The first knowledge-base plugin produced end-to-end by `engineering/book-to-skill`, from OpenAI's +[Spinning Up in Deep RL](https://spinningup.openai.com/) (MIT, © 2018 OpenAI; primarily developed by +Joshua Achiam). Source obtained by cloning `openai/spinningup` and compiling its `docs/` reStructuredText +tree — 38 files, ~37k words, ~49K tokens — through the full pipeline: `extract_document.py --mode technical` +→ analysis → 20 chapter files → glossary / patterns / cheatsheet → master `SKILL.md` → `book_skill_validator.py` +→ `skill_plugin_emitter.py`. Validator passes clean **in `--strict` mode**; every file is inside budget +(resident core 2,066 / 4,000 tokens; 20 chapters averaging ~1,256 tokens each, loaded on demand). + +- **Rights basis is `open-license`, not fair use.** The emitter's Step-11 gate refuses a shareable package + without one; MIT permits derivative distribution. Upstream's notice is reproduced in full in the plugin's + `LICENSE` alongside this package's own, and the README names the source, the author and the commit-era — + a sidecar JSON is not a license notice. +- **Chapter structure follows the source's own `toctree`**, not a heading scan: user documentation (ch01-06), + Introduction to RL Parts 1-3 (ch07-09), resources — the researcher essay, key papers, exercises, benchmarks + (ch10-13), one chapter per algorithm in lineage order (ch14-19: VPG → TRPO → PPO, DDPG → TD3 → SAC), and the + logger/MPI/ExperimentGrid utilities (ch20). +- **The cheatsheet carries what a glossary cannot**: the debug-turnaround threshold (under 5 minutes), the seed + minimums (3, ideally 10+), the benchmark network defaults that differ by algorithm family ((64,32)/tanh + on-policy vs (256,256)/relu off-policy), and Spinning Up's own **parity disclosure** — DDPG/TD3/SAC are + research-grade, VPG/TRPO/PPO are not, and the docs say to use OpenAI Baselines for those instead. +- **One emitter defect found and fixed in the same change.** `skill_plugin_emitter.py` was writing its `source` + provenance block into `plugin.json`, which this repo's own `scripts/check_plugin_json.py` hard-fails + (issue #954 — Claude Code rejects the whole manifest on any unrecognized key). Its inline comment claimed + `source`/`attribution` were "approved extension fields," which had been true and no longer was. The manifest + builder now emits spec fields only and a new `_authoring_notes()` writes `.claude-plugin/authoring-notes.json`. + Every future `book-to-skill` package gets a manifest that passes CI; the one emitted here was the first to + catch it. +- **write-a-skill checklist: 4/6, and the two remaining are structural, not defects.** A compiled + knowledge base is not a hand-authored instruction skill. (2) "SKILL.md under 100 lines" conflicts + with the format's mandatory 20-row Chapter Index plus Topic Index; the binding budget here is + `book_skill_validator.py`'s 4,000-token cap, and the file sits at 2,101 — trimming to 100 lines + would delete exactly the navigation that `budget.over_cap` exists to protect. (3) "no + time-sensitive info" fires on "January 2020", which is a **provenance pin on a frozen source** + (Spinning Up's last update), not a staleness claim — removing it would make the skill less honest + about what it covers. (5) was a real gap and is fixed: the resident core now carries a worked + invocation block. +- **No new Python tools** — a compiled knowledge base ships notes, not scripts, so the tools and references + counters are unchanged. **Counters:** skills 386 → 387; agents 116 → 117; commands 146 → 147; plugins + 97 → 98 (verified via `scripts/derive_counters.py --check`). + +--- + **Unreleased (post-v2.11.2) — engineering/skill-doctor (grade the agent setup from real sessions):** Rebuild of [warpdotdev/common-skills](https://github.com/warpdotdev/common-skills)' `skill-doctor` (MIT, © Denver Technologies, Inc., pinned at `f3b58c81`) as a full plugin. Harvests the last N days of local Claude Code / Codex sessions scoped to one repo, has the agent judge each condensed transcript against two verbatim-preserved rubrics (efficiency, code quality — labels only, closed tables), measures which installed skills actually fired, and proposes only the skill edits the evidence justifies — one local, self-contained HTML report; nothing is ever uploaded. @@ -205,7 +249,7 @@ Rebuild of [warpdotdev/common-skills](https://github.com/warpdotdev/common-skill Derived from [virgiliojr94/book-to-skill](https://github.com/virgiliojr94/book-to-skill) (MIT). Compiles a book, documentation folder, or spec collection (PDF, EPUB, DOCX, HTML, Markdown, RST, AsciiDoc, RTF, MOBI/AZW) into an agent skill: a resident master `SKILL.md` (core frameworks + chapter index + topic index, capped at 4k tokens) plus on-demand `chapters/chNN-*.md`, `glossary.md`, `patterns.md`, and a decision `cheatsheet.md`. The agent reads the core, then one chapter — never the whole source again. - **Vendored close to verbatim:** the extraction library (`scripts/book_to_skill/` — config, exceptions, sanitize, dependencies, utils + 7 per-format parsers) keeps upstream's format chains, chapter detection across Latin/Roman/Chinese/Thai/Korean heading styles, invisible-Unicode (Trojan Source) sanitization and DOCX entity-expansion guard. -- **25 numbered deviations from upstream** — the list in `engineering/book-to-skill/README.md` is authoritative; the plugin's `authoring-notes.json` `attribution.derivation_note` summarizes it. Highlights: (5) `--install-missing` now defaults to `report` — it prints the pip command and uses the stdlib fallback instead of upstream's TTY prompt that runs `pip install` into the caller's environment; (6) a **rights gate** — `skill_plugin_emitter.py --distribution shareable` refuses without `--rights` from `public-domain|open-license|internal-docs|author-permission`, with `fair-use` deliberately excluded (a defence, not a licence); (10) the two upstream validators merged into one four-family gate, adding **budget** and **index** families — dead chapter links, unindexed chapter files and dangling topic refs are the failure that silently breaks navigation while the skill still looks complete, and upstream had no check for it; (11) folded YAML scalars now parse, so a wrapped description no longer under-reports its length past the 1024-char cap; (12) `discovery_tax.py` → `token_budget_estimator.py` with the optional `tiktoken` path dropped, a post-flight budget audit added, and an explicit **worth-converting verdict** that says "just read it" when the source is under ~3× the compiled skill. +- **26 numbered deviations from upstream** — the list in `engineering/book-to-skill/README.md` is authoritative; the plugin's `authoring-notes.json` `attribution.derivation_note` summarizes it. Highlights: (5) `--install-missing` now defaults to `report` — it prints the pip command and uses the stdlib fallback instead of upstream's TTY prompt that runs `pip install` into the caller's environment; (6) a **rights gate** — `skill_plugin_emitter.py --distribution shareable` refuses without `--rights` from `public-domain|open-license|internal-docs|author-permission`, with `fair-use` deliberately excluded (a defence, not a licence); (10) the two upstream validators merged into one four-family gate, adding **budget** and **index** families — dead chapter links, unindexed chapter files and dangling topic refs are the failure that silently breaks navigation while the skill still looks complete, and upstream had no check for it; (11) folded YAML scalars now parse, so a wrapped description no longer under-reports its length past the 1024-char cap; (12) `discovery_tax.py` → `token_budget_estimator.py` with the optional `tiktoken` path dropped, a post-flight budget audit added, and an explicit **worth-converting verdict** that says "just read it" when the source is under ~3× the compiled skill. - **Repo-native addition with no upstream counterpart — Step 11 / `/cs:book-to-plugin`:** upstream stops at a bare folder in `~/.claude/skills/`, which this library cannot route to. `skill_plugin_emitter.py` wraps a compiled skill as a full plugin package (manifest + `cs-` agent + `/cs:` command + README) and prints the marketplace entry; it never edits `marketplace.json` itself, and refuses to wrap a skill carrying validation errors. - **Cross-linked into `engineering/write-a-skill`** ("author first, compile second" — that skill authors from expertise in your head, this one compiles from a document on disk). - 4 stdlib-only tools (all `--help` / `--sample` / `--output json`), 5 references citing 7–8 sources each, 3 assets, `cs-book-to-skill` agent, 2 commands. **Counters:** skills 362 → 363; tools 644 → 663; refs 741 → 746; agents 102 → 103; commands 116 → 118; plugins 88 → 89 (derived via `scripts/derive_counters.py --check`). @@ -732,4 +776,4 @@ When I correct you, or you catch yourself making a mistake: before continuing ad **Last Updated:** August 24, 2026 **Version:** v2.12.0 (consolidated release — see CHANGELOG.md) -**Status:** 386 production-ready skills across 20 domains, 97 marketplace plugins, docs site live (counters derived via `scripts/derive_counters.py`) +**Status:** 387 production-ready skills across 20 domains, 98 marketplace plugins, docs site live (counters derived via `scripts/derive_counters.py`) diff --git a/README.md b/README.md index 7ddf98e3..518ab55d 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ # Claude Code Skills & Plugins — Agent Skills for Every Coding Tool -**386 production-ready Claude Code skills, plugins, and agent skills for 13 AI coding tools.** +**387 production-ready Claude Code skills, plugins, and agent skills for 13 AI coding tools.** The most comprehensive open-source library of Claude Code skills and agent plugins — also works with OpenAI Codex, Gemini CLI, Cursor, and 9 more coding agents. Reusable expertise packages covering engineering, DevOps, marketing (incl. AEO — Answer Engine Optimization for LLM citation), security (PreToolUse hooks), compliance, C-level advisory (incl. founder-mode CFO/CMO/CRO/CPO/COO/CHRO/CISO/GC/CDO/CAIO/CCO/VPE personas + 21 /cs:* slash commands), productivity (capture/email/reflect/weekly-review/deep-work/meetings), an academic research stack (litreview/grants/dossier/patent/syllabus/pulse/notebooklm/deep-research + hybrid router), and enterprise Research Operations (clinical-research/research-finance/market-research/product-research, v2.9.0). @@ -10,10 +10,10 @@ The most comprehensive open-source library of Claude Code skills and agent plugi [^vibe]: Mistral Vibe is also **BYO-sync tier**: the repo ships a pre-generated `.vibe/skills/claude-skills/` tree, run `./scripts/vibe-install.sh` once locally to install into `~/.vibe/skills/`. Same agentskills.io SKILL.md standard — no format conversion. Docs: . [![License: MIT](https://img.shields.io/badge/License-MIT-yellow?style=for-the-badge)](https://opensource.org/licenses/MIT) -[![Skills](https://img.shields.io/badge/Skills-386-brightgreen?style=for-the-badge)](#skills-overview) -[![Agents](https://img.shields.io/badge/Agents-116-blue?style=for-the-badge)](#agents) +[![Skills](https://img.shields.io/badge/Skills-387-brightgreen?style=for-the-badge)](#skills-overview) +[![Agents](https://img.shields.io/badge/Agents-117-blue?style=for-the-badge)](#agents) [![Personas](https://img.shields.io/badge/Personas-7-purple?style=for-the-badge)](#personas) -[![Commands](https://img.shields.io/badge/Commands-146-orange?style=for-the-badge)](#commands) +[![Commands](https://img.shields.io/badge/Commands-147-orange?style=for-the-badge)](#commands) [![Stars](https://img.shields.io/github/stars/alirezarezvani/claude-skills?style=for-the-badge)](https://github.com/alirezarezvani/claude-skills/stargazers) [![SkillCheck Validated](https://img.shields.io/badge/SkillCheck-Validated-4c1?style=for-the-badge)](https://getskillcheck.com) @@ -152,12 +152,12 @@ Run `./scripts/convert.sh --tool all` to generate tool-specific outputs locally. ## Skills Overview -**386 production-ready skills across 20 domains:** +**387 production-ready skills across 20 domains:** | Domain | Skills | Highlights | Details | |--------|--------|------------|---------| | **🔧 Engineering — Core** | 53 | Architecture, frontend, backend, fullstack, QA, DevOps, SecOps, AI/ML, data, Playwright Pro (test gen, flaky fix, migrations), self-improving agent (auto-memory curation), security suite, a11y audit, **named-persona-adversarial-review** (review via named engineering philosophies), **embedded-iot-mentor** (MCU/board selection, firmware-reuse-first, breadboard-MVP discipline) | [engineering-team/](engineering-team/) | -| **⚡ Engineering — POWERFUL** | 91 | Agent designer, RAG architect, database designer, CI/CD builder, security auditor, MCP builder, AgentHub, Helm charts, Terraform, self-eval, llm-wiki, tc-tracker, autoresearch-agent, **reliability portfolio** (feature-flags-architect, kubernetes-operator, chaos-engineering, slo-architect), ship-gate, security-guidance PreToolUse hook, **Matt Pocock skills** (write-a-skill, caveman, grill-me, handoff, grill-with-docs), **zero-hallucination-coder** (Discuss→Map→Decompose→Execute→Verify), **agent-harness** (goal→plan→execute→verify→close loops over any domain), **memory-engineering** (price the memory write path, pick which cost to pay, audit FACT/SKILL/LOG density, gate on a forgetting policy), **skillopt-sleep** (nightly gated self-evolution from real Claude Code sessions, vendored from microsoft/SkillOpt), **book-to-skill** (compile a book, docs folder, or spec collection into a knowledge-base skill, then package it as a plugin), **boost-asio-pro** (async C++ networking — version-gated coroutine/callback styles, strand discipline), **human-gate** (batched human review as a structured artifact + a gate that refuses to close on open blockers), **agent-memory** (four-tier L0-L3 memory ladder over Claude Code hooks; promotion earned by recurrence across sessions and days, redacted or contested claims refuse to promote, nothing reaches CLAUDE.md without a human adopt), **skill-doctor** (grade the agent setup from real session history — rubric-judged, deterministically gated, evidence-traced skill edits) | [engineering/](engineering/) | +| **⚡ Engineering — POWERFUL** | 92 | Agent designer, RAG architect, database designer, CI/CD builder, security auditor, MCP builder, AgentHub, Helm charts, Terraform, self-eval, llm-wiki, tc-tracker, autoresearch-agent, **reliability portfolio** (feature-flags-architect, kubernetes-operator, chaos-engineering, slo-architect), ship-gate, security-guidance PreToolUse hook, **Matt Pocock skills** (write-a-skill, caveman, grill-me, handoff, grill-with-docs), **zero-hallucination-coder** (Discuss→Map→Decompose→Execute→Verify), **agent-harness** (goal→plan→execute→verify→close loops over any domain), **memory-engineering** (price the memory write path, pick which cost to pay, audit FACT/SKILL/LOG density, gate on a forgetting policy), **skillopt-sleep** (nightly gated self-evolution from real Claude Code sessions, vendored from microsoft/SkillOpt), **book-to-skill** (compile a book, docs folder, or spec collection into a knowledge-base skill, then package it as a plugin), **boost-asio-pro** (async C++ networking — version-gated coroutine/callback styles, strand discipline), **human-gate** (batched human review as a structured artifact + a gate that refuses to close on open blockers), **agent-memory** (four-tier L0-L3 memory ladder over Claude Code hooks; promotion earned by recurrence across sessions and days, redacted or contested claims refuse to promote, nothing reaches CLAUDE.md without a human adopt), **skill-doctor** (grade the agent setup from real session history — rubric-judged, deterministically gated, evidence-traced skill edits), **spinning-up-deep-rl** (knowledge base compiled from OpenAI's Spinning Up in Deep RL — RL fundamentals, the model-free taxonomy, policy-gradient derivations, and one chapter per reference algorithm) | [engineering/](engineering/) | | **🎯 Product** | 17 | Product manager, agile PO, strategist, UX researcher, UI design, landing pages, SaaS scaffolder, analytics, experiment designer, discovery, roadmap communicator, code-to-prd, apple-hig-expert | [product-team/](product-team/) | | **📣 Marketing** | 49 | 8 pods: Content, SEO + AEO (`aeo` — E-E-A-T audit, citation tracking across 5 LLMs) + local (`local-seo-manager` — GBP/NAP/Map-Pack), CRO, Channels, Growth, Intelligence, Sales + `business-name-fit` (cross-cultural naming) + context foundation + orchestration router | [marketing-skill/](marketing-skill/) | | **🚀 Productivity** | 12 | `capture` (brain-dump-to-action), `email` pair (inbox-setup + inbox-triage), `reflect` (journal), `handoff` (Matt Pocock-inspired), `andreessen` (market-first decision mode), `roast` (5-angle idea panel → GO/RESHAPE/KILL), `fable-goal` (ramble → autonomous /goal prompt), `weekly-review` (GTD loop with refusal gate), `deep-work` (time-blocking + shallow-work budget), `meetings` (cost gate + agenda + action items), `swedish-mentor` (CEFR-leveled Swedish learning paths) | [productivity/](productivity/) | diff --git a/engineering/book-to-skill/.claude-plugin/authoring-notes.json b/engineering/book-to-skill/.claude-plugin/authoring-notes.json index 41270b08..b1d7127f 100644 --- a/engineering/book-to-skill/.claude-plugin/authoring-notes.json +++ b/engineering/book-to-skill/.claude-plugin/authoring-notes.json @@ -3,6 +3,6 @@ "derived_from": "https://github.com/virgiliojr94/book-to-skill", "original_author": "virgiliojr94", "original_license": "MIT", - "derivation_note": "The extraction library (book_to_skill/ — config, exceptions, sanitize, dependencies, utils, and the 7 per-format parsers) is vendored from upstream and carries upstream's multi-format chains, chapter detection across 5 script families, and Unicode/XXE hardening. 25 numbered deviations are recorded in README.md, which is the authoritative list: the workflow was rewritten Claude-Code-first for this repo's conventions, the 3 upstream tools were reworked into 4 argparse CLIs with --help/--sample, install-on-import behaviour was replaced with a report-only default, and Step 11 (plugin emission with a rights gate) plus the validator's budget and index families are additions with no upstream counterpart." + "derivation_note": "The extraction library (book_to_skill/ — config, exceptions, sanitize, dependencies, utils, and the 7 per-format parsers) is vendored from upstream and carries upstream's multi-format chains, chapter detection across 5 script families, and Unicode/XXE hardening. 26 numbered deviations are recorded in README.md, which is the authoritative list: the workflow was rewritten Claude-Code-first for this repo's conventions, the 3 upstream tools were reworked into 4 argparse CLIs with --help/--sample, install-on-import behaviour was replaced with a report-only default, and Step 11 (plugin emission with a rights gate) plus the validator's budget and index families are additions with no upstream counterpart." } } diff --git a/engineering/book-to-skill/README.md b/engineering/book-to-skill/README.md index 076dc790..c695e42c 100644 --- a/engineering/book-to-skill/README.md +++ b/engineering/book-to-skill/README.md @@ -101,7 +101,7 @@ small; just read it." ## Deviations from upstream -**This numbered list is the authoritative record.** `plugin.json`'s +**This numbered list is the authoritative record.** `.claude-plugin/authoring-notes.json`'s `attribution.derivation_note` summarizes it; if the two ever disagree, this list wins. **Structural** @@ -245,9 +245,10 @@ small; just read it." preventing hand-edit mistakes contradicted the manifest beside it); the symlink guard is backed by a **post-copy re-walk** that deletes the package if a link appeared during the copy, closing the check-then-act window rather than only narrowing it; and the manifest - carries `source.license_scope` stating that the top-level `license` covers the package + emits `source.license_scope` stating that the top-level `license` covers the package scaffolding, not the compiled notes — a distinction that previously lived only in README - prose where a tool reading the manifest alone would miss it. + prose where a tool reading the metadata alone would miss it. (Deviation 26 later moved + that whole `source` block out of `plugin.json` and into the sidecar.) 21. **The documented quick-start actually runs.** SKILL.md's copy-paste block referenced `$WORKDIR` and `$SKILLS_HOME` without ever defining them — following it literally produced @@ -297,6 +298,21 @@ small; just read it." leaves the victim's contents intact and is replaced by a 0600 file we own. Degrades to the previous path-based checks on platforms without `dir_fd`/`O_NOFOLLOW` (Windows). +26. **Provenance moved out of `plugin.json` into the sidecar the repo actually allows.** + The manifest builder wrote its whole `source` block (spec, build pattern, source document, + chapter count, distribution, `license_scope`, `rights_basis`) into `plugin.json`, and an + inline comment asserted that `source` and `attribution` were "approved extension fields." + That had been true and no longer was: Claude Code rejects an entire manifest on any + unrecognized key (issue #954), and this repo's own `scripts/check_plugin_json.py` hard-fails + a `plugin.json` carrying either field, pointing at `.claude-plugin/authoring-notes.json` + instead. So every package the emitter produced failed the repo's blocking CI gate the moment + it was committed — a defect that only surfaces at the very last step of the pipeline, which + is why it survived. `_plugin_manifest()` now emits spec fields only, and a new + `_authoring_notes()` writes the `source` block to `.claude-plugin/authoring-notes.json`. + Note that `source` remains a *valid* key in a `marketplace.json` `plugins[]` entry, which is + how it leaked into the manifest in the first place; the printed marketplace snippet is + unchanged and still correct. + --- ## Security audit diff --git a/engineering/book-to-skill/skills/book-to-skill/scripts/skill_plugin_emitter.py b/engineering/book-to-skill/skills/book-to-skill/scripts/skill_plugin_emitter.py index 17b7f7a3..bc11b54b 100644 --- a/engineering/book-to-skill/skills/book-to-skill/scripts/skill_plugin_emitter.py +++ b/engineering/book-to-skill/skills/book-to-skill/scripts/skill_plugin_emitter.py @@ -166,13 +166,13 @@ def _assert_no_symlinks(skill_dir: Path) -> None: def _plugin_manifest(identity: dict, *, domain: str, author: str, author_url: str, - repository: str, distribution: str, rights: str | None, - source_note: str) -> dict: - # Key set and the two approved extension fields (`source`, `attribution`) are - # fixed by scripts/check_plugin_json.py. Anything else — including a - # top-level "private" flag — fails the repo's CI gate, so the distribution - # marker lives inside `source`. - manifest = { + repository: str) -> dict: + # Spec fields ONLY. Claude Code's manifest validator rejects the whole + # plugin.json on any unrecognized key (issue #954), and the repo's own gate + # (scripts/check_plugin_json.py) hard-fails a manifest carrying `source` or + # `attribution`. Provenance therefore goes to the sidecar file the validator + # never reads — see _authoring_notes below. + return { "name": identity["slug"], "description": identity["description"], "version": "1.0.0", @@ -185,31 +185,42 @@ def _plugin_manifest(identity: dict, *, domain: str, author: str, author_url: st # the manifest sees the distinction that README.md makes in prose. "license": "MIT", "skills": [f"./skills/{identity['slug']}"], - "source": { - "spec": "generated by engineering/book-to-skill", - "build_pattern": "book-to-skill conversion: extract -> analyze -> chapter files " - "+ glossary + patterns + cheatsheet -> master SKILL.md", - "distinct_from": "not a hand-authored skill; every claim traces to the source " - "document named below", - "source_document": source_note or identity["title"], - "chapters": identity["chapters"], - "distribution": distribution, - "license_scope": ( - "The top-level `license` covers this package's scaffolding only. The " - "compiled notes under skills/ are derived from the source document and " - "carry that work's terms; see source.rights_basis." - ), - }, + } + + +def _authoring_notes(identity: dict, *, distribution: str, rights: str | None, + source_note: str) -> dict: + """Provenance for `.claude-plugin/authoring-notes.json`. + + The repo's schema allows exactly two keys here, `source` and `attribution`; + a generated skill's provenance is a `source` block. This file is authoring + metadata that Claude Code's manifest validator never reads, which is + precisely why the fields live here rather than in plugin.json. + """ + source = { + "spec": "generated by engineering/book-to-skill", + "build_pattern": "book-to-skill conversion: extract -> analyze -> chapter files " + "+ glossary + patterns + cheatsheet -> master SKILL.md", + "distinct_from": "not a hand-authored skill; every claim traces to the source " + "document named below", + "source_document": source_note or identity["title"], + "chapters": identity["chapters"], + "distribution": distribution, + "license_scope": ( + "plugin.json's top-level `license` covers this package's scaffolding only. " + "The compiled notes under skills/ are derived from the source document and " + "carry that work's terms; see source.rights_basis." + ), } if rights: - manifest["source"]["rights_basis"] = rights - manifest["source"]["rights_note"] = RIGHTS_BASES[rights] + source["rights_basis"] = rights + source["rights_note"] = RIGHTS_BASES[rights] if distribution == "local": # A local package is study notes from a work the user owns. The flag is # advisory metadata, not enforcement — it exists so a later publish step # (or a reviewer) can see the package was never cleared for sharing. - manifest["source"]["cleared_for_distribution"] = False - return manifest + source["cleared_for_distribution"] = False + return {"source": source} def _agent_markdown(identity: dict, domain: str, source_note: str) -> str: @@ -407,10 +418,12 @@ def emit(*, skill_dir: Path, dest_root: Path, domain: str, author: str, author_u _assert_replaceable(package_root, resolved_dest, skill_dir) manifest = _plugin_manifest(identity, domain=domain, author=author, author_url=author_url, - repository=repository, distribution=distribution, rights=rights, - source_note=source_note) + repository=repository) + notes = _authoring_notes(identity, distribution=distribution, rights=rights, + source_note=source_note) files = { ".claude-plugin/plugin.json": json.dumps(manifest, indent=2, ensure_ascii=False) + "\n", + ".claude-plugin/authoring-notes.json": json.dumps(notes, indent=2, ensure_ascii=False) + "\n", "README.md": _readme_markdown(identity, domain=domain, distribution=distribution, rights=rights, source_note=source_note), f"agents/cs-{identity['slug']}.md": _agent_markdown(identity, domain, source_note), diff --git a/engineering/spinning-up-deep-rl/.claude-plugin/authoring-notes.json b/engineering/spinning-up-deep-rl/.claude-plugin/authoring-notes.json new file mode 100644 index 00000000..57efee25 --- /dev/null +++ b/engineering/spinning-up-deep-rl/.claude-plugin/authoring-notes.json @@ -0,0 +1,13 @@ +{ + "source": { + "spec": "generated by engineering/book-to-skill", + "build_pattern": "book-to-skill conversion: extract -> analyze -> chapter files + glossary + patterns + cheatsheet -> master SKILL.md", + "distinct_from": "not a hand-authored skill; every claim traces to the source document named below", + "source_document": "Spinning Up in Deep RL by Joshua Achiam (OpenAI)", + "chapters": 20, + "distribution": "shareable", + "license_scope": "plugin.json's top-level `license` covers this package's scaffolding only. The compiled notes under skills/ are derived from the source document and carry that work's terms; see source.rights_basis.", + "rights_basis": "open-license", + "rights_note": "the source work carries a licence permitting derivative distribution" + } +} diff --git a/engineering/spinning-up-deep-rl/.claude-plugin/plugin.json b/engineering/spinning-up-deep-rl/.claude-plugin/plugin.json new file mode 100644 index 00000000..f2851a73 --- /dev/null +++ b/engineering/spinning-up-deep-rl/.claude-plugin/plugin.json @@ -0,0 +1,15 @@ +{ + "name": "spinning-up-deep-rl", + "description": "Knowledge base from \"Spinning Up in Deep RL\" by Joshua Achiam (OpenAI, MIT-licensed). Use when applying Achiam's frameworks for RL fundamentals and MDPs, the model-free algorithm taxonomy, policy gradient derivations, the six reference algorithms (VPG, TRPO, PPO, DDPG, TD3, SAC), debugging silently-failing RL code, or running rigorous multi-seed RL experiments.", + "version": "1.0.0", + "author": { + "name": "Alireza Rezvani", + "url": "https://github.com/alirezarezvani" + }, + "homepage": "https://github.com/alirezarezvani/claude-skills/tree/main/engineering/spinning-up-deep-rl", + "repository": "https://github.com/alirezarezvani/claude-skills", + "license": "MIT", + "skills": [ + "./skills/spinning-up-deep-rl" + ] +} diff --git a/engineering/spinning-up-deep-rl/LICENSE b/engineering/spinning-up-deep-rl/LICENSE new file mode 100644 index 00000000..ec66b3fc --- /dev/null +++ b/engineering/spinning-up-deep-rl/LICENSE @@ -0,0 +1,58 @@ +This package has two layers with two different terms. + +------------------------------------------------------------------------------ +1. Package scaffolding — plugin manifest, agent, command, README +------------------------------------------------------------------------------ + +MIT License + +Copyright (c) 2026 Alireza Rezvani + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +THE SOFTWARE. + +------------------------------------------------------------------------------ +2. Compiled notes under skills/spinning-up-deep-rl/ +------------------------------------------------------------------------------ + +Derived from OpenAI's "Spinning Up in Deep RL" documentation +(https://spinningup.openai.com/ · https://github.com/openai/spinningup), +primarily developed by Joshua Achiam. Distributed under the same MIT terms, +whose notice is reproduced below in full as that licence requires. + +The MIT License + +Copyright (c) 2018 OpenAI (http://openai.com) + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +THE SOFTWARE. diff --git a/engineering/spinning-up-deep-rl/README.md b/engineering/spinning-up-deep-rl/README.md new file mode 100644 index 00000000..46818d89 --- /dev/null +++ b/engineering/spinning-up-deep-rl/README.md @@ -0,0 +1,49 @@ +# Spinning Up in Deep RL + +Knowledge-base plugin compiled from **Spinning Up in Deep RL by Joshua Achiam (OpenAI)** by +[`engineering/book-to-skill`](../book-to-skill/). 20 chapters indexed. + +## What is in here + +| File | Contents | +|------|----------| +| `skills/spinning-up-deep-rl/SKILL.md` | Core frameworks, chapter index, topic index (resident, under 4k tokens) | +| `skills/spinning-up-deep-rl/chapters/` | One summary per chapter — loaded on demand, never all at once | +| `skills/spinning-up-deep-rl/glossary.md` | Every significant term, alphabetized, with its chapter | +| `skills/spinning-up-deep-rl/patterns.md` | Techniques and design patterns with trade-offs | +| `skills/spinning-up-deep-rl/cheatsheet.md` | Decision rules, thresholds and trade-off matrices | + +## Use + +``` +/cs:spinning-up-deep-rl # core frameworks + chapter index +/cs:spinning-up-deep-rl # resolve via topic index, read one chapter +/cs:spinning-up-deep-rl ch05 # read one chapter summary +``` + +Or invoke the `cs-spinning-up-deep-rl` agent for a working session anchored to this source. + +## Provenance and limits + +**Source:** OpenAI's [Spinning Up in Deep RL](https://spinningup.openai.com/) +([openai/spinningup](https://github.com/openai/spinningup)), primarily developed by +**Joshua Achiam**. Compiled from the `docs/` reStructuredText tree at the January 2020 +PyTorch update. + +**Rights basis:** `open-license`. The source is **MIT, Copyright (c) 2018 OpenAI**, which +permits derivative distribution. The full upstream notice is reproduced in +[`LICENSE`](LICENSE) alongside this package's own; the top-level `license` field in +`plugin.json` covers the scaffolding only. + +Generated, not hand-authored: every claim traces to the source document. It carries that source's +blind spots, and it is a set of structured notes — **not a copy of the work and not a substitute +for reading it**. + +**What it does not cover:** DQN and the discrete-action value-learning family, recurrent or +convolutional architectures, partially-observed settings, model-based implementations, and any +deep RL work after early 2020. The six implementations documented are educational; `ch13` records +which are research-grade (DDPG, TD3, SAC) and which are not (VPG, TRPO, PPO). + +Distribution: `shareable`. Regenerate or extend with +`python3 engineering/book-to-skill/skills/book-to-skill/scripts/extract_document.py`, then re-run +`book_skill_validator.py` before loading the result. diff --git a/engineering/spinning-up-deep-rl/agents/cs-spinning-up-deep-rl.md b/engineering/spinning-up-deep-rl/agents/cs-spinning-up-deep-rl.md new file mode 100644 index 00000000..08efbafa --- /dev/null +++ b/engineering/spinning-up-deep-rl/agents/cs-spinning-up-deep-rl.md @@ -0,0 +1,37 @@ +--- +name: cs-spinning-up-deep-rl +description: Answers from the knowledge base compiled from Spinning Up in Deep RL by Joshua Achiam (OpenAI). Loads the master frameworks first and reads a single chapter file on demand rather than the whole source. Refuses to answer beyond what the source covers. +skills: engineering/spinning-up-deep-rl/skills/spinning-up-deep-rl +domain: engineering +model: opus +tools: [Read, Grep, Glob] +--- + +# Spinning Up in Deep RL — Knowledge Agent + +## Voice + +**Opening:** "Which framework or chapter are you reaching for?" +**Forcing question:** "Is this something the source actually covers, or are you asking me to +extrapolate past it?" +**Closing:** "That is the author's formulation, from ch. Anything past it is my inference, not theirs." + +## Purpose + +Applies the frameworks compiled from **Spinning Up in Deep RL by Joshua Achiam (OpenAI)** (20 chapters +indexed) while the user works. Answers with the author's exact naming, then cites the chapter. + +## How it navigates + +1. Read `skills/spinning-up-deep-rl/SKILL.md` — Core Frameworks and both indexes. +2. Match the question against the Topic Index; read **only** the chapter files it points to. +3. Reach for `glossary.md` for a term, `patterns.md` for a technique, `cheatsheet.md` for a decision. +4. Never load every chapter — that is the cost this skill exists to avoid. + +## Hard rules + +- **Cite the chapter.** Every framework claim names the chapter it came from. +- **Do not extrapolate silently.** If the source does not cover it, say so before answering from + general knowledge, and label which is which. +- **Preserve exact naming.** The author's term is the interface; a paraphrase breaks lookup. +- **Do not reproduce the source at length.** These are structured notes, not a copy of the work. diff --git a/engineering/spinning-up-deep-rl/commands/cs-spinning-up-deep-rl.md b/engineering/spinning-up-deep-rl/commands/cs-spinning-up-deep-rl.md new file mode 100644 index 00000000..3b82679f --- /dev/null +++ b/engineering/spinning-up-deep-rl/commands/cs-spinning-up-deep-rl.md @@ -0,0 +1,30 @@ +--- +name: "cs-spinning-up-deep-rl" +description: "/cs:spinning-up-deep-rl [topic | framework | chNN] — query the knowledge base compiled from Spinning Up in Deep RL by Joshua Achiam (OpenAI). Use when applying its frameworks while working, looking up a term, or reading one chapter's summary." +--- + +# /cs:spinning-up-deep-rl — Spinning Up in Deep RL + +**Command:** `/cs:spinning-up-deep-rl [topic | framework name | chNN]` + +## When to run + +- Applying a framework from this source to work in progress +- Looking up the author's exact formulation of a term +- Reading one chapter's compiled summary without opening the source +- Checking whether the source covers a question at all + +## What it does + +1. Loads `engineering/spinning-up-deep-rl/skills/spinning-up-deep-rl/SKILL.md` — Core Frameworks plus the Chapter and Topic indexes. +2. **No argument** → reports the core frameworks and the chapter index. +3. **A topic or framework name** → resolves it through the Topic Index and reads only the + matching chapter file. +4. **`chNN`** → reads that chapter's summary directly. +5. Answers with the author's naming and cites the chapter. + +## Boundary + +This command answers from **one source** (20 chapters indexed). Anything it does not +cover gets said out loud rather than filled in — and hands-on work in your codebase belongs to the +engineering skills, not here. diff --git a/engineering/spinning-up-deep-rl/skills/spinning-up-deep-rl/SKILL.md b/engineering/spinning-up-deep-rl/skills/spinning-up-deep-rl/SKILL.md new file mode 100644 index 00000000..7314e535 --- /dev/null +++ b/engineering/spinning-up-deep-rl/skills/spinning-up-deep-rl/SKILL.md @@ -0,0 +1,192 @@ +--- +name: spinning-up-deep-rl +description: "Knowledge base from \"Spinning Up in Deep RL\" by Joshua Achiam (OpenAI, MIT-licensed). Use when applying Achiam's frameworks for RL fundamentals and MDPs, the model-free algorithm taxonomy, policy gradient derivations, the six reference algorithms (VPG, TRPO, PPO, DDPG, TD3, SAC), debugging silently-failing RL code, or running rigorous multi-seed RL experiments." +--- + +# Spinning Up in Deep RL + +**Author**: Joshua Achiam (OpenAI) | **Source**: spinningup.readthedocs.io, MIT | **Chapters**: 20 | **Generated**: 2026-08-25 + +## How to Use This Skill + +- **No argument** — load the core frameworks below +- **A topic** — ask about `advantage function`, `target networks`, `entropy regularization`; + I resolve it through the Topic Index and read that chapter file +- **`chNN`** — I load that chapter's summary +- **"what chapters do you have?"** — the full index + +``` +/cs:spinning-up-deep-rl # core frameworks + chapter index +/cs:spinning-up-deep-rl entropy regularization # topic index -> ch19, read that chapter +/cs:spinning-up-deep-rl ch09 # one chapter summary +``` + +When you ask about something not in Core Frameworks, I read the relevant chapter file before +answering rather than guessing from the index. + +--- + +## Core Frameworks & Mental Models + +### The RL problem +`pi* = argmax_pi J(pi)`, where `J(pi) = E_{tau~pi}[R(tau)]`. Every algorithm approximates +this; where it substitutes a different objective (a Bellman residual, a surrogate), that +substitution is the source of its failure modes. Four value functions — `V^pi`, `Q^pi`, `V*`, +`Q*` — all obey Bellman self-consistency, and `a*(s) = argmax_a Q*(s,a)` is why Q-learning +is a viable family at all. Advantage `A^pi(s,a) = Q^pi(s,a) - V^pi(s)` is the relative-quality +signal policy gradients run on. (ch07) + +### The two branching questions +Place any algorithm by asking: **does it have or learn a model**, and **what does it learn** +(policy, Q-function, value function, model). That generates the whole landscape. (ch08) + +### Policy optimization vs Q-learning — the central trade-off +- **Policy optimization** is *principled: you directly optimize the thing you want.* Stable + and reliable. On-policy, so it cannot reuse data, so it is sample-hungry. +- **Q-learning** only *indirectly* optimizes performance, by training `Q_theta` to satisfy a + self-consistency equation. Many failure modes, so less stable. But substantially more + sample efficient when it works, because it reuses everything. +- **Satisfying the Bellman equations well carries no guarantee of good policy performance.** +- The two are not exclusive — DDPG and SAC live between them deliberately. (ch08) + +### The policy gradient template +`grad J = E[ sum_t grad log pi_theta(a_t|s_t) * Phi_t ]`. Five valid choices of `Phi_t`: +full return, reward-to-go, reward-to-go minus a baseline, `Q^pi`, and `A^pi`. All share an +expectation and differ in variance. Two rules get you from the first to the last: +- **Don't let the past distract you** — drop rewards obtained before the action. Those terms + had zero mean and nonzero variance: pure noise. +- **Baselines** — by the EGLP lemma, any state-only `b(s)` can be added or subtracted freely. + The standard choice is `V^pi(s_t)`, learned by MSE regression onto reward-to-go. (ch09) + +### The policy-gradient loss is not a loss function +Its data distribution depends on the parameters, and it does not measure performance even in +expectation. Only at the current parameters, with data from those parameters, does it have +the negative gradient of performance. **You can send it to negative infinity while performance +craters, and it usually will. Only average return means anything.** (ch09) + +### Broken RL code almost always fails silently +It runs fine; the agent just never learns. Usually something is computed with the wrong +equation, on the wrong distribution, or piped to the wrong place. **If it doesn't work, assume +there's a bug** before touching hyperparameters. Debug by measuring everything and reading the +code critically. The archetype is one missing `squeeze`: a `[N]` vs `[N,1]` shape mismatch is +broadcast-compatible, raises nothing, and silently turns the Bellman backup into an `[N,N]` +matrix. (ch10, ch12) + +### Learn by doing +Write your own implementations, **shortest correct version** of each, **simplest algorithms +first**. VPG, DQN, A2C, PPO, DDPG, roughly in that order; ~250-300 lines each. Single-threaded +before parallel. **Iterate fast in simple environments — under 5 minutes turnaround at the +debug stage.** Do not attempt Atari or Humanoid before the toy task works. Read papers for +their ablations and supplementary material, but **do not overfit to paper details** (the +original DDPG's architecture, init scheme and batch norm are not strictly necessary) **or to +existing implementations** (their abstractions serve reuse, not your single use case). (ch10) + +### Rigor: four standards +1. **Fair comparisons** — tune the baseline as hard as your method; never handicap it. +2. **Remove stochasticity as a confounder** — at least 3 seeds, 10 or more to be thorough. + Two seed groups can produce curves that look like different distributions. +3. **High-integrity experiments** — launch fresh final runs and precommit to reporting them. + Tuning produces hypotheses; final runs produce conclusions. +4. **Check each claim separately** — ablate every design decision. (ch10) + +### The safe-step family (on-policy) +VPG takes an unconstrained gradient step, so a single bad step can collapse performance. +**TRPO** constrains the step in **KL-divergence between policies, not distance in parameter +space**, then backtracking-line-searches until the exact constraint holds. **PPO** drops the +constraint and instead **clips the objective so the policy gains nothing by moving far**, +which is first-order, far simpler, and empirically at least as good. (ch14, ch15, ch16) + +### The overestimation family (off-policy) +DDPG amortizes the intractable continuous `max_a Q(s,a)` into a learned policy: +`max_a Q(s,a) ~= Q(s, mu(s))`. It needs a **replay buffer** (licensed because the Bellman +equation is indifferent to how data was collected) and **target networks** (because the target +otherwise depends on the parameters being trained). Its failure mode is **Q-value +overestimation, which the policy actively exploits**. **TD3** answers with clipped double-Q, +delayed policy updates and target policy smoothing. **SAC** adds **entropy regularization**, +making the explore-exploit trade-off an explicit coefficient `alpha`. (ch17, ch18, ch19) + +--- + +## Chapter Index + +| # | Title | Key Frameworks | +|---|-------|----------------| +| [ch01](chapters/ch01-introduction.md) | Introduction | The missing middle step, Code Design Philosophy | +| [ch02](chapters/ch02-installation.md) | Installation | Install-then-verify, MuJoCo optionality | +| [ch03](chapters/ch03-algorithm-lineup.md) | Algorithms: What's Included and Why | The two lineages, on/off-policy trade-off, code template | +| [ch04](chapters/ch04-running-experiments.md) | Running Experiments | One flag per kwarg, ExperimentGrid, save-dir suffixes | +| [ch05](chapters/ch05-experiment-outputs.md) | Experiment Outputs | Tools not files, watch-then-measure | +| [ch06](chapters/ch06-plotting-results.md) | Plotting Results | `Performance` alias, prefix autocompletion, seed averaging | +| [ch07](chapters/ch07-key-concepts-in-rl.md) | Part 1: Key Concepts in RL | MDPs, four value functions, Bellman equations, advantage | +| [ch08](chapters/ch08-kinds-of-rl-algorithms.md) | Part 2: Kinds of RL Algorithms | Taxonomy, model bias, policy-opt vs Q-learning | +| [ch09](chapters/ch09-intro-to-policy-optimization.md) | Part 3: Intro to Policy Optimization | Log-derivative trick, EGLP lemma, reward-to-go, baselines | +| [ch10](chapters/ch10-spinning-up-as-a-researcher.md) | Spinning Up as a Deep RL Researcher | Learn by doing, three idea frames, four rigor standards | +| [ch11](chapters/ch11-key-papers-in-deep-rl.md) | Key Papers in Deep RL | 13-section topic map | +| [ch12](chapters/ch12-exercises.md) | Exercises | Problem Set 1 and 2, the silent DDPG bug | +| [ch13](chapters/ch13-benchmarks.md) | Benchmarks | The parity disclosure, family-specific metrics | +| [ch14](chapters/ch14-vpg.md) | Vanilla Policy Gradient | The six-step loop | +| [ch15](chapters/ch15-trpo.md) | Trust Region Policy Optimization | KL trust region, line search, conjugate gradient | +| [ch16](chapters/ch16-ppo.md) | Proximal Policy Optimization | PPO-Clip, KL early stopping | +| [ch17](chapters/ch17-ddpg.md) | Deep Deterministic Policy Gradient | MSBE, replay buffers, target networks, polyak | +| [ch18](chapters/ch18-td3.md) | Twin Delayed DDPG | Clipped double-Q, delayed updates, target smoothing | +| [ch19](chapters/ch19-sac.md) | Soft Actor-Critic | Entropy regularization, reparameterization, squashed Gaussian | +| [ch20](chapters/ch20-logger-and-utilities.md) | Logger, MPI Tools and Run Utils | EpochLogger pattern, MPI PyTorch order | + +## Topic Index + +- **Advantage function** ch07, ch09, ch14 +- **Baselines** ch09 +- **Bellman equations** ch07, ch17 +- **Benchmarks / parity** ch13, ch01 +- **Clipped double-Q** ch18, ch19 +- **Continuous action spaces** ch07, ch17 +- **Debugging / silent failure** ch10, ch12 +- **DDPG** ch17, ch03, ch08 +- **Entropy regularization** ch19 +- **Exploration vs exploitation** ch14, ch17, ch19 +- **GAE** ch09, ch14 +- **Installation** ch02 +- **KL divergence / trust region** ch15, ch16 +- **Logging** ch20, ch05 +- **MDPs** ch07 +- **Model-based RL** ch08 +- **MPI / parallelization** ch20, ch02, ch04 +- **MSBE** ch17 +- **Off-policy** ch03, ch08, ch17 +- **On-policy** ch03, ch08, ch14 +- **Papers / literature** ch11, ch10 +- **Plotting** ch06, ch13 +- **Policies (categorical, Gaussian, squashed)** ch07, ch19 +- **Policy gradient derivation** ch09 +- **PPO** ch16, ch03 +- **Q-learning** ch08, ch07 +- **Replay buffer** ch17 +- **Reparameterization trick** ch19, ch10 +- **Research process / rigor** ch10, ch13 +- **Reward-to-go** ch09 +- **Running experiments** ch04, ch05 +- **SAC** ch19, ch03, ch08 +- **Seeds / variance** ch10, ch13, ch04 +- **Target networks / polyak** ch17, ch18 +- **TD3** ch18, ch12 +- **TRPO** ch15, ch03 +- **Value functions** ch07, ch09 +- **VPG** ch14, ch09 + +## Supporting Files + +- [glossary.md](glossary.md) · [patterns.md](patterns.md) · [cheatsheet.md](cheatsheet.md) + +## Scope & Limits + +Covers the Spinning Up documentation only, as of the January 2020 PyTorch update. It does +**not** cover: DQN and the discrete-action value-learning family (referenced, never +implemented here), recurrent or convolutional architectures, partially-observed settings, +model-based implementations, exploration/meta-RL/hierarchy beyond ch11's reading list, or any +deep RL work after early 2020. The six implementations are educational; ch13 says which are +research-grade. For topics beyond this source, I say so rather than improvising. + +--- + +*Compiled from OpenAI's Spinning Up in Deep RL documentation (MIT, Copyright (c) 2018 OpenAI), +primarily developed by Joshua Achiam. Structured study notes, not a reproduction of the source.* diff --git a/engineering/spinning-up-deep-rl/skills/spinning-up-deep-rl/chapters/ch01-introduction.md b/engineering/spinning-up-deep-rl/skills/spinning-up-deep-rl/chapters/ch01-introduction.md new file mode 100644 index 00000000..3af4c15b --- /dev/null +++ b/engineering/spinning-up-deep-rl/skills/spinning-up-deep-rl/chapters/ch01-introduction.md @@ -0,0 +1,102 @@ +# Chapter 1: Introduction + +## Core Idea +Spinning Up exists to be the missing middle step between "I want to work on deep RL" +and "I can implement and evaluate a deep RL algorithm" — the field has no standard +textbook, papers omit the design details that decide whether code works, and production +RL libraries hide the algorithm inside framework abstractions. + +## Frameworks Introduced +- **The missing middle step**: the gap Spinning Up fills, between high-level awareness + (what topics exist, why they matter) and the ability to transmute an algorithm into code. + - When to use: deciding whether a resource teaches you deep RL or merely describes it. + - How: judge a resource on whether it closes the theory-to-code gap. Papers give theory + and omit details; libraries give code that hides the algorithm. Neither closes it alone. + +- **Code Design Philosophy** (the four rules the implementations obey): + - **As simple as possible while still being reasonably good.** Not state of the art; + good enough to reach roughly the intended performance. + - **Highly consistent with each other**, so understanding one makes the next painless. + - **Almost completely self-contained** — virtually no shared code between algorithms + except logging, saving, loading and MPI utilities. You can study one algorithm without + following a chain of dependencies. + - **Patterned to come as close to pseudocode as possible**, minimizing the theory-code gap. + - How: when reading or writing RL code, prefer the version you can read top to bottom + over the version that reuses the most code. Reuse is an engineering virtue that is a + pedagogical cost. + +- **Minimize tricks and minimize differences**: the implementations deliberately omit + tricks present in the original papers (e.g. the regularization terms in the original + Soft Actor-Critic code, observation normalization in all algorithms) and deliberately + remove gratuitous differences between similar algorithms (DDPG, TD3 and SAC all run + gradient updates after fixed intervals of environment interaction, so they compare cleanly). + - When to use: any time you are comparing two algorithms and want the comparison to be + about the algorithms rather than about their implementation choices. + +## Key Concepts +- **Deep RL**: reinforcement learning (learning to solve tasks by trial and error) combined + with deep learning. +- **Reasonably good**: achieves roughly the intended performance but does not necessarily + match the best reported results in the literature on every task. +- **Maintenance mode**: Spinning Up's current status. Breaking bugs get repaired; no major + new features are planned. +- **The Rosetta Stone goal**: the motivation for the January 2020 PyTorch update — the same + algorithm expressed in two neural network libraries so the algorithm is separable from the + library. + +## Mental Models +- Think of an RL library's abstraction layer as a **tax on learning**: good for code reuse + between algorithms, unnecessary if you are writing one algorithm for one use case. +- Use "**can I read this algorithm without opening another file?**" as the test of whether + an implementation is written to be learned from. +- Think of the resource landscape as **two failure modes**: papers that obscure key design + details, and public implementations that are hard to read. Spinning Up targets both. + +## Anti-patterns +- **Using Spinning Up's implementations for scientific benchmarking comparisons**: they are + "reasonably good," not best-reported. See ch13 for which ones are at parity (DDPG, TD3, SAC) + and which are not (VPG, TRPO, PPO). Use OpenAI Baselines for TRPO/PPO research comparisons. +- **Assuming an educational implementation is a research implementation**: the omitted tricks + (observation normalization, normalized value regression targets) are exactly what separates + the two. +- **Treating deep RL as engineering-only**: the material explicitly serves people from + professions with no connection to engineering or computer science who nonetheless need to + make informed decisions about the technology. + +## Worked Example +Support history, read as a record of what the community actually asked for after release: + +| Date | Event | +|------|-------| +| Nov 8, 2018 | Initial release, followed by three weeks of high-bandwidth support | +| April 2019 | Six-month internal review of community feedback | +| Jan 2020 | The PyTorch update ships | +| Future | No major updates planned | + +The April 2019 review surfaced exactly three requests, in priority order: +1. **Implementations in other neural network libraries** — enough people had written their + own PyTorch ports (Fired Up, Spinning Up Basic, Torching Up) that a "Rosetta Stone for + deep RL" became the top priority. This one shipped. +2. **Open source RL environments** (e.g. PyBullet) for benchmarks, examples and exercises, + to avoid the proprietary MuJoCo dependency. Did not ship. +3. **More algorithms**, especially Deep Q-Networks. Did not ship. + +The lesson for anyone building an educational resource: the community's top request was not +more content, it was the same content expressed in the framework they already use. + +## Key Takeaways +1. The barrier to entry in deep RL is not intelligence or math; it is that theory and code + are documented in separate places and neither is written to connect to the other. +2. Self-contained beats DRY when the reader is trying to learn the algorithm. +3. Consistency across implementations is a teaching feature: differences that remain are + real algorithmic differences. +4. "Reasonably good" is a deliberate, stated performance target — check ch13 before citing + any Spinning Up number as a benchmark. +5. Deep RL is central to AI safety work in OpenAI's framing; the resource exists partly as + a recruiting and capability-building pipeline for that. + +## Connects To +- **Ch 3**: which algorithms were chosen and why, and the code template they all share. +- **Ch 10**: the essay on becoming a researcher — the practice counterpart to this chapter's + philosophy. +- **Ch 13**: the benchmark numbers that qualify "reasonably good." diff --git a/engineering/spinning-up-deep-rl/skills/spinning-up-deep-rl/chapters/ch02-installation.md b/engineering/spinning-up-deep-rl/skills/spinning-up-deep-rl/chapters/ch02-installation.md new file mode 100644 index 00000000..e8d35068 --- /dev/null +++ b/engineering/spinning-up-deep-rl/skills/spinning-up-deep-rl/chapters/ch02-installation.md @@ -0,0 +1,88 @@ +# Chapter 2: Installation + +## Core Idea +Spinning Up needs Python 3, OpenAI Gym and OpenMPI, on Linux or OSX; MuJoCo is optional +but is the de facto benchmarking standard, and everything in the book works without it on +the free Classic Control and Box2D environments. + +## Frameworks Introduced +- **Install then immediately verify with a real training run**: the install is not confirmed + by a successful `pip install`, it is confirmed by an agent visibly learning. + - When to use: every fresh environment, before debugging anything else. + - How: run the install test below, then watch the policy and plot the curve. Three + commands, three different subsystems (training, rendering, plotting) verified. + +## Key Concepts +- **OpenMPI**: the message-passing library used for the parallelized (on-policy) algorithms. + Installed with `apt-get install libopenmpi-dev` on Ubuntu, `brew install openmpi` on Mac. +- **MuJoCo**: a proprietary physics engine; free to trial and free for full-time students, + otherwise paid. The de facto standard for benchmarking continuous-control deep RL. +- **mujoco-py**: the Python bindings that let Gym use MuJoCo. +- **Classic Control / Box2D**: Gym environment families that are completely free and + sufficient to get started. +- **Editable install**: `pip install -e .` from the cloned repo, so edits to the algorithm + files take effect without reinstalling — the intended workflow for a repo you are meant + to read and modify. + +## Code Examples +Environment and dependencies: + +```bash +conda create -n spinningup python=3.6 +conda activate spinningup + +# Ubuntu +sudo apt-get update && sudo apt-get install libopenmpi-dev +# Mac OS X (requires Homebrew) +brew install openmpi + +git clone https://github.com/openai/spinningup.git +cd spinningup +pip install -e . +``` + +Verify the install (roughly 10 minutes; leave it running and keep reading): + +```bash +python -m spinup.run ppo --hid "[32,32]" --env LunarLander-v2 \ + --exp_name installtest --gamma 0.999 +python -m spinup.run test_policy data/installtest/installtest_s0 +python -m spinup.run plot data/installtest/installtest_s0 +``` + +Optional MuJoCo, after following the mujoco-py README and obtaining a license: + +```bash +pip install gym[mujoco,robotics] +python -m spinup.run ppo --hid "[32,32]" --env Walker2d-v2 --exp_name mujocotest +``` + +## Reference Tables + +| Requirement | Status | Notes | +|-------------|--------|-------| +| Python 3 | Required | Anaconda recommended; the docs pin 3.6 | +| OpenAI Gym | Required | Installed by `pip install -e .`, MuJoCo envs excluded | +| OpenMPI | Required | Needed for the MPI-parallelized on-policy algorithms | +| Linux / OSX | Required | Windows unsupported and untested; one community workaround exists | +| MuJoCo | Optional | Proprietary, licensed; preferred because of benchmarking convention | + +## Anti-patterns +- **Skipping the install test** and then debugging an algorithm that was never installed + correctly. The install test costs ten unattended minutes. +- **Treating a MuJoCo licence as a prerequisite for learning RL**: Classic Control and Box2D + are free and adequate for everything up to benchmarking. +- **Fighting package management ad hoc**: the docs go out of their way to point at conda + explainers, because "I just installed this thing but it says it's not found" is the + predicted failure, not an unusual one. + +## Key Takeaways +1. Install is Python 3 + Gym + OpenMPI, plus optional MuJoCo; Linux or OSX only. +2. `pip install -e .` (editable) because you are expected to modify the code. +3. Verification means an actual short PPO run, a rendered policy and a plot. +4. Not having MuJoCo blocks benchmarking, not learning. + +## Connects To +- **Ch 4**: `python -m spinup.run` is the entry point every later chapter uses. +- **Ch 13**: the benchmarks that make MuJoCo the convention. +- **Ch 20**: the MPI utilities that OpenMPI enables. diff --git a/engineering/spinning-up-deep-rl/skills/spinning-up-deep-rl/chapters/ch03-algorithm-lineup.md b/engineering/spinning-up-deep-rl/skills/spinning-up-deep-rl/chapters/ch03-algorithm-lineup.md new file mode 100644 index 00000000..dce7b34f --- /dev/null +++ b/engineering/spinning-up-deep-rl/skills/spinning-up-deep-rl/chapters/ch03-algorithm-lineup.md @@ -0,0 +1,101 @@ +# Chapter 3: Algorithms — What's Included and Why + +## Core Idea +The six algorithms (VPG, TRPO, PPO, DDPG, TD3, SAC) are not a survey; they are two +lineages of ideas, each starting from a foundational algorithm and progressively fixing +its defining weakness, culminating in PPO and SAC. + +## Frameworks Introduced +- **The two lineages**: + - **On-policy line: VPG to TRPO to PPO.** Each step buys back sample efficiency without + giving up the stability that comes from directly optimizing the objective you care about. + - **Off-policy line: DDPG to TD3 and SAC.** Each step mitigates the brittleness that comes + from optimizing a Bellman self-consistency condition instead of performance itself. + - When to use: placing any new algorithm you read about. Ask which line it is on and which + weakness of its predecessor it claims to fix. + +- **The central trade-off the lineup exposes**: on-policy algorithms *directly optimize the + objective you care about* — policy performance — and it works out mathematically that + you need on-policy data to compute the updates, so they cannot reuse old data. Off-policy + algorithms exploit the Bellman optimality equations, which hold for *any* transition data, + so they reuse everything — but satisfying Bellman's equations well carries **no guarantee** + of good policy performance. + - How: choose the family by which risk you can afford. Stability with a sample budget, or + sample efficiency with a tuning budget. + +- **The standard two-file code template**: every implementation splits into an *algorithm + file* (experience buffer class, then one function that runs the algorithm) and a *core + file* (utilities, the actor-critic constructors, the MLP actor-critic). + - When to use: reading any Spinning Up algorithm, or structuring your own. + +## Key Concepts +- **On-policy**: each update uses only data collected by the most recent version of the policy. +- **Off-policy**: each update can use data collected at any point in training, however the + agent was exploring at the time. +- **MLP actor-critic**: all six use non-recurrent multi-layer-perceptron actor-critics, which + makes them suitable for fully-observed, non-image-based environments (e.g. Gym MuJoCo) and + unsuitable, as shipped, for partial observability or pixels. +- **Sample efficiency**: how much environment interaction is needed to reach a performance level. +- **Deterministic policy gradients**: the theory (2014) that made DDPG possible — much younger + than the policy gradient theory behind VPG, whose core elements go back to the late 80s. + +## Reference Tables + +| Algorithm | Family | Policy | Action spaces | Parallel (MPI) | PyTorch | TF1 | +|-----------|--------|--------|---------------|----------------|---------|-----| +| VPG | On-policy | Stochastic | Discrete + continuous | Yes | Yes | Yes | +| TRPO | On-policy | Stochastic | Discrete + continuous | Yes | No | Yes | +| PPO | On-policy | Stochastic | Discrete + continuous | Yes | Yes | Yes | +| DDPG | Off-policy | Deterministic | Continuous only | No | Yes | Yes | +| TD3 | Off-policy | Deterministic | Continuous only | No | Yes | Yes | +| SAC | Off-policy | Stochastic | Continuous only (as shipped) | No | Yes | Yes | + +## Code Examples +The PyTorch algorithm function, in order — the template every implementation follows: + +``` +1) Logger setup +2) Random seed setting +3) Environment instantiation +4) Build the actor-critic module via the `actor_critic` function passed in as an argument +5) Instantiate the experience buffer +6) Set up callable loss functions that also return algorithm-specific diagnostics +7) Make PyTorch optimizers +8) Set up model saving through the logger +9) Set up an update function: one epoch of optimization, or one step of descent +10) Main loop: a) run the agent in the environment + b) periodically update parameters per the algorithm's main equations + c) log key performance metrics and save the agent +``` + +The TF1 version is the same shape with four graph-construction steps inserted (placeholders, +actor-critic graph, loss/diagnostic graph, training ops) and a session step. + +## Mental Models +- Read the lineup as **"progressions of ideas from the recent history of the field"**, not as + a menu. VPG predates deep RL entirely; DDPG's theory is from 2014. +- Treat **PPO and SAC as the two defaults** — they are close to state of the art on + reliability and sample efficiency among policy-learning algorithms. +- Think of the `actor_critic` argument as the seam: the algorithm function is fixed, the + network construction is injected. That is also where the ch12 silent bug lives. + +## Anti-patterns +- **Reaching for DDPG on a discrete action space.** DDPG, TD3 and the shipped SAC are + continuous-only. The max over actions is why (see ch17). +- **Expecting these to work on Atari or partially-observed tasks as shipped**: MLP, + non-recurrent, non-image. +- **Picking an algorithm by recency**: TD3 and SAC were published roughly concurrently; SAC + is not a successor to TD3, it is a parallel branch that borrowed the clipped double-Q trick. + +## Key Takeaways +1. Two families, one trade-off: directly optimize performance (on-policy, stable, sample-hungry) + or exploit Bellman (off-policy, sample-efficient, no performance guarantee). +2. PPO and SAC are the practical end points of each line. +3. Continuous-only for the whole off-policy line, as implemented here. +4. Every implementation is two files and the same ten-step function; learn the template once. +5. TRPO has no PyTorch implementation in Spinning Up. + +## Connects To +- **Ch 8**: the full taxonomy this lineup is a slice of. +- **Ch 14-19**: one chapter per algorithm, in lineage order. +- **Ch 12**: the exercises that make the on-policy/off-policy distinction concrete. diff --git a/engineering/spinning-up-deep-rl/skills/spinning-up-deep-rl/chapters/ch04-running-experiments.md b/engineering/spinning-up-deep-rl/skills/spinning-up-deep-rl/chapters/ch04-running-experiments.md new file mode 100644 index 00000000..2d0366ac --- /dev/null +++ b/engineering/spinning-up-deep-rl/skills/spinning-up-deep-rl/chapters/ch04-running-experiments.md @@ -0,0 +1,133 @@ +# Chapter 4: Running Experiments + +## Core Idea +`spinup/run.py` turns every keyword argument of every algorithm into a command-line flag, +and turns a list of values for a flag into a series of experiments — so hyperparameter +sweeps and multi-seed runs are one command, not a script. + +## Frameworks Introduced +- **One flag per kwarg**: if `kwarg` is a valid keyword argument of an algorithm function, + `--kwarg` sets it. `python -m spinup.run [algo] --help` prints the docstring readout. + - How: `python -m spinup.run [algo name] [experiment flags]`. + +- **Multiple values means multiple experiments**: providing more than one value for an + argument launches one experiment per combination, **in series**. They do not run in + parallel because a single experiment already soaks up enough resources that concurrency + buys no speedup. + - When to use: seed sweeps (the ch10 rigor requirement) and hyperparameter ablations. + +- **ExperimentGrid**: the in-script equivalent, based on but simpler than rllab's + VariantGenerator. `spinup.run` uses one under the hood. + - How: `eg.add(param_name, values, shorthand, in_name)` then `eg.run(thunk, **run_kwargs)`. + `in_name` forces a parameter into the experiment name even when it does not vary. + - Difference from the CLI: no shortcut kwargs — you must write `ac_kwargs:hidden_sizes`, + not `hid`. + +- **Shorthand-driven save directories**: results land in + `data_dir/[outer_prefix]exp_name[suffix]/[inner_prefix]exp_name[suffix]_s[seed]`. + The suffix encodes only the hyperparameters that *differ across the launched experiments*, + and never the seed — so runs that differ only by seed group into the same folder, which is + exactly what the plotter needs to average over seeds. + +## Key Concepts +- **`--env` / `--env_name`**: a Gym environment name, converted internally into the `env_fn` + callable every algorithm actually takes. +- **`--hid` / `--act`**: shortcut flags for `ac_kwargs:hidden_sizes` and `ac_kwargs:activation`, + valid for all current algorithms. +- **`--cpu` / `--num_cpu`**: launch with this many MPI-connected processes; `auto` uses all + available. Raises an error for algorithms that do not support parallelization. +- **`--dt` / `--datestamp`**: put timestamps in the save directory names. +- **User-supplied shorthand**: square brackets after a flag, e.g. `--hid[h]`, control the + directory-name abbreviation. Without one, a shorthand is derived automatically + (`clip_ratio` becomes `cli`). +- **`eval()` passthrough**: flag values pass through `eval()` before use, so you can name + functions and objects directly, e.g. `--act torch.nn.ELU`. +- **Dict kwargs**: `--key:v1 value_1 --key:v2 value_2` instead of `--key dict(v1=..., v2=...)`. + +## Code Examples +The quickstart, with every flag doing something distinct: + +```bash +python -m spinup.run ppo --exp_name ppo_ant --env Ant-v2 --clip_ratio 0.1 0.2 \ + --hid[h] [32,32] [64,32] --act torch.nn.Tanh --seed 0 10 20 --dt \ + --data_dir path/to/data +``` + +Choosing the backend explicitly: + +```bash +python -m spinup.run ppo_pytorch --env Walker2d-v2 --exp_name walker +python -m spinup.run ppo_tf1 --env Walker2d-v2 --exp_name walker +# bare `ppo` reads spinup/user_config.py for the default backend +``` + +From a script: + +```python +from spinup import ppo_pytorch as ppo +import gym +env_fn = lambda: gym.make('LunarLander-v2') +ac_kwargs = dict(hidden_sizes=[64, 64]) +logger_kwargs = dict(output_dir='path/to/output_dir', exp_name='experiment_name') +ppo(env_fn=env_fn, ac_kwargs=ac_kwargs, steps_per_epoch=5000, epochs=250, + logger_kwargs=logger_kwargs) +``` + +ExperimentGrid, from `spinup/examples/pytorch/bench_ppo_cartpole.py`: + +```python +from spinup.utils.run_utils import ExperimentGrid +from spinup import ppo_pytorch +import torch + +eg = ExperimentGrid(name='ppo-pyt-bench') +eg.add('env_name', 'CartPole-v0', '', True) +eg.add('seed', [10*i for i in range(args.num_runs)]) +eg.add('epochs', 10) +eg.add('steps_per_epoch', 4000) +eg.add('ac_kwargs:hidden_sizes', [(32,), (64,64)], 'hid') +eg.add('ac_kwargs:activation', [torch.nn.Tanh, torch.nn.ReLU], '') +eg.run(ppo_pytorch, num_cpu=args.cpu) +``` + +## Worked Example +`python -m spinup.run ddpg_tf1 --env Hopper-v2 --hid[h] [300] [128,128] --act tf.nn.tanh tf.nn.relu` + +Two `hid` values times two `act` values is four experiments, run in series, producing four +suffixes: + +``` +_h128-128_ac-actrelu +_h128-128_ac-acttanh +_h300_ac-actrelu +_h300_ac-acttanh +``` + +`h` came from the user-supplied `[h]`; `ac-act` was derived automatically from the true flag +name `ac_kwargs:activation`. Add `--seed 0 10 20` and you get twelve runs in four folders — +three seeds grouped per configuration, which is the grouping the plotter averages over. + +## Anti-patterns +- **Running the per-algorithm files directly** (`spinup/algos/BACKEND/ALGO/ALGO.py`). The + command-line support there is vestigial, takes a different argument set, and is explicitly + not the recommended way to run experiments. +- **Using ZShell without escaping square brackets**: ZShell treats them as special characters + and Spinning Up uses them for both list values and shorthands. +- **One seed.** RL algorithms have high variance; the flag exists so you use it (ch10 asks + for at least 3, ideally 10 or more). +- **Expecting parallel experiment launches.** Sweeps run in series by design. + +## Key Takeaways +1. Every kwarg is a flag; every list of flag values is a sweep. +2. Sweeps run in series; MPI parallelism is per-experiment via `--cpu`, and only for + algorithms that support it. +3. Save-directory suffixes encode only the varying hyperparameters, never the seed — that + grouping is what makes multi-seed plotting work. +4. `ExperimentGrid` is the scripted form and loses only the shortcut kwargs. +5. Values go through `eval()`, so activations and other objects can be named on the command line. + +## Connects To +- **Ch 5**: what the directories those flags create actually contain. +- **Ch 6**: the plotter, which consumes this directory structure and its autocompletion. +- **Ch 10**: why multi-seed runs are a rigor requirement, not a convenience. +- **Ch 20**: `ExperimentGrid`, `call_experiment` and `setup_logger_kwargs` in Run Utils. diff --git a/engineering/spinning-up-deep-rl/skills/spinning-up-deep-rl/chapters/ch05-experiment-outputs.md b/engineering/spinning-up-deep-rl/skills/spinning-up-deep-rl/chapters/ch05-experiment-outputs.md new file mode 100644 index 00000000..1784a74e --- /dev/null +++ b/engineering/spinning-up-deep-rl/skills/spinning-up-deep-rl/chapters/ch05-experiment-outputs.md @@ -0,0 +1,97 @@ +# Chapter 5: Experiment Outputs + +## Core Idea +Every run saves four things — a config record, a TSV of training metrics, the trained +model, and a pickled copy of the environment — and only one of them (`config.json`) is +ever meant to be read by hand. + +## Frameworks Introduced +- **Tools, not files**: `test_policy` loads from `pyt_save/` or `tf1_save/`, the plotter + interprets `progress.txt`. Those are the correct interfaces. `config.json` is the only + file you should ever open yourself, and only to remember what you ran. + - When to use: any time you are tempted to parse a save directory manually. + +- **Watch, then measure**: `test_policy` renders the agent so you can see behavior that no + scalar metric would have shown you; `-nr` drops rendering when you only want the numbers. + +## Key Concepts +- **`progress.txt`**: tab-separated records of every metric the logger recorded, e.g. `Epoch`, + `AverageEpRet`. +- **`config.json`**: as-complete-as-possible dict of the args and kwargs used to launch the run. + Record-keeping only — launching an experiment from a config file is not supported. +- **`vars.pkl`**: algorithm state; currently used only to save a copy of the environment. +- **`pyt_save/model.pt`**: a pickled PyTorch `nn.Module`; loading restores an ActorCritic + object with an `act` method. +- **`tf1_save/`**: `variables/`, `model_info.pkl` (key-to-tensor-name map) and `saved_model.pb`. + Renamed from `simple_save/` on 2020-01-30. +- **`DEFAULT_DATA_DIR`**: set in `spinup/user_config.py`; defaults to `spinningup/data`. + +## Reference Tables + +| File | Contents | +|------|----------| +| `pyt_save/` | PyTorch only. Everything needed to restore the agent and value functions | +| `tf1_save/` | TF1 only. SavedModel plus the key-to-tensor map | +| `config.json` | The launch args and kwargs. Non-serializable values become strings | +| `progress.txt` | TSV of logged metrics across training | +| `vars.pkl` | Pickled environment copy; may be empty if the env cannot be pickled | + +`test_policy` flags: + +| Flag | Default | Does | +|------|---------|------| +| `-l L`, `--len=L` | 0 | Max episode length; 0 means no maximum | +| `-n N`, `--episodes=N` | 100 | Number of test episodes | +| `-nr`, `--norender` | off | Print returns and lengths only; much faster | +| `-i I`, `--itr=I` | -1 | Which saved snapshot; -1 is latest (see below) | +| `-d`, `--deterministic` | off | **SAC only.** Use the deterministic mean policy | + +## Code Examples +```bash +python -m spinup.run test_policy path/to/output_directory +``` + +When the environment failed to pickle and `test_policy` raises `AssertionError: Environment +not found!`, rebuild it by hand: + +```python +from spinup.utils.test_policy import load_policy_and_env, run_policy +import your_env +_, get_action = load_policy_and_env('/path/to/output_directory') +env = your_env.make() +run_policy(env, get_action) +``` + +To keep snapshots from many points in training (off by default — algorithms overwrite the +most recent one), change the line present in every algorithm: + +```python +logger.save_state({'env': env}, None) # default: one snapshot, overwritten +logger.save_state({'env': env}, epoch) # keep per-epoch snapshots +``` + +and then set `save_freq` to something reasonable — at the default of 1 you flood the output +directory with one folder per epoch. + +## Anti-patterns +- **Forgetting `-d` when evaluating SAC.** SAC trains a stochastic policy but the correct + evaluation policy is the deterministic mean. Without the flag you are measuring the wrong + thing, and the flag is used for no other algorithm. +- **Expecting to resume training.** Spinning Up implementations have no way to resume a + partially-trained agent. +- **Trying to relaunch from `config.json`.** It is a record, not an input. +- **Assuming `vars.pkl` has your environment.** Gym Box2D environments in older Gym versions + are known to fail to pickle, leaving it empty. + +## Key Takeaways +1. Four artifacts per run; use the tools for three of them and read only `config.json` by hand. +2. SAC evaluation requires `--deterministic`; every other algorithm ignores it. +3. Multi-snapshot saving is a two-line change plus a `save_freq` you must set deliberately. +4. Environment pickling can silently fail — the recovery path is rebuilding the env in Python. +5. No resume support; a killed run is a lost run. + +## Connects To +- **Ch 4**: the flags that create these directories. +- **Ch 6**: `progress.txt` is what the plotter reads. +- **Ch 19**: why SAC's evaluation policy differs from its training policy. +- **Ch 20**: the EpochLogger that writes all of this. diff --git a/engineering/spinning-up-deep-rl/skills/spinning-up-deep-rl/chapters/ch06-plotting-results.md b/engineering/spinning-up-deep-rl/skills/spinning-up-deep-rl/chapters/ch06-plotting-results.md new file mode 100644 index 00000000..9b83646c --- /dev/null +++ b/engineering/spinning-up-deep-rl/skills/spinning-up-deep-rl/chapters/ch06-plotting-results.md @@ -0,0 +1,79 @@ +# Chapter 6: Plotting Results + +## Core Idea +The plotter reads a directory prefix, autocompletes every matching run beneath it, averages +across seeds by default, and resolves the pseudo-metric `Performance` to the *correct* +performance measure for each algorithm family — so a fair on-policy versus off-policy +comparison is one command. + +## Frameworks Introduced +- **`Performance` as a family-aware alias**: `Performance` is not a real output of any + algorithm. The plotter resolves it per logdir to `AverageEpRet` for on-policy algorithms + and `AverageTestEpRet` for off-policy ones. + - When to use: always, unless you specifically want one raw column. It is the default `-y`. + - Why it matters: on-policy performance is the average return of the batch just collected; + off-policy performance is measured by separate deterministic test rollouts. Plotting the + same raw column for both would compare two different quantities. + +- **Prefix autocompletion**: logdirs are searched recursively and prefixes expand. Give the + plotter `data/bench_algo` and it finds `bench_algo1` and `bench_algo2` with all their seeds. + - How: name experiments with a shared prefix at launch and comparison becomes free. + +- **Average by default, `--count` to disaggregate**: by default, y-values are averaged across + all results sharing an `exp_name` — typically identical experiments differing only in seed. + `--count` shows each curve separately. + +## Key Concepts +- **`--xaxis` / `-x`**: which column is the x-axis. Default `TotalEnvInteracts`, i.e. sample + efficiency is the default framing, not wall-clock or epochs. +- **`--value` / `-y`**: which columns to graph; multiple values produce multiple graphs. +- **`--smooth S`**: average over a fixed window of width S. The book's own benchmark plots + use a window of 11 epochs (ch13). +- **`--legend` / `-l`**: overrides the automatic legend, which uses `exp_name` from + `config.json`. One string per *matched* directory, which may exceed the number of logdir + arguments you passed. +- **`--select` / `--exclude`**: keep or drop curves whose logdir contains all of / any of + these substrings. The way to narrow an over-eager autocomplete. + +## Code Examples +```bash +python -m spinup.run plot [path/to/output_directory ...] [--legend [LEGEND ...]] \ + [--xaxis XAXIS] [--value [VALUE ...]] [--count] [--smooth S] \ + [--select [SEL ...]] [--exclude [EXC ...]] +``` + +Comparing two algorithms across all their seeds, relying on autocompletion: + +``` +data/ + bench_algo1/ + bench_algo1-seed0/ + bench_algo1-seed10/ + bench_algo2/ + bench_algo2-seed0/ + bench_algo2-seed10/ +``` + +```bash +python spinup/utils/plot.py data/bench_algo +``` + +## Anti-patterns +- **Passing one legend string per logdir argument.** Autocompletion may match several + directories per argument; you need one legend entry per match, unless you narrowed the + set with `--select` / `--exclude`. +- **Plotting `AverageEpRet` for an off-policy algorithm** and calling it performance — that + is the behavior policy with exploration noise, not the evaluated policy. Use `Performance`. +- **Heavy smoothing to make a result look clean.** Smoothing is a display parameter; it does + not change what the seeds did, and ch10 asks you to show seed variance rather than hide it. + +## Key Takeaways +1. `Performance` is the correct default because it is algorithm-family aware. +2. Prefix autocompletion plus shared `exp_name` prefixes is the intended comparison workflow. +3. Seed averaging is on by default; `--count` reveals the individual curves that average hides. +4. The default x-axis is environment interactions — sample efficiency is the default question. + +## Connects To +- **Ch 5**: `progress.txt`, the plotter's input. +- **Ch 4**: the suffix rules that group seeds into one folder for averaging. +- **Ch 13**: the benchmark plots, produced with 10 seeds and an 11-epoch smoothing window. diff --git a/engineering/spinning-up-deep-rl/skills/spinning-up-deep-rl/chapters/ch07-key-concepts-in-rl.md b/engineering/spinning-up-deep-rl/skills/spinning-up-deep-rl/chapters/ch07-key-concepts-in-rl.md new file mode 100644 index 00000000..6457c4b6 --- /dev/null +++ b/engineering/spinning-up-deep-rl/skills/spinning-up-deep-rl/chapters/ch07-key-concepts-in-rl.md @@ -0,0 +1,139 @@ +# Chapter 7: Part 1 — Key Concepts in RL + +## Core Idea +RL is the study of agents learning by trial and error; formally, the agent maximizes +expected return J(pi) over trajectories, and almost every algorithm gets there through one +of four value functions, all of which obey Bellman self-consistency equations. + +## Frameworks Introduced +- **The agent-environment interaction loop**: at every step the agent sees a (possibly + partial) observation, chooses an action, and receives a reward. The environment changes + because of the action, and may also change on its own. + - How: name the five MDP pieces before writing any code — states S, actions A, reward + function R, transition function P, start-state distribution rho_0. + +- **The RL optimization problem**: `pi* = argmax_pi J(pi)`, where + `J(pi) = E_{tau ~ pi}[R(tau)]` and the trajectory distribution is + `P(tau|pi) = rho_0(s_0) * prod_t P(s_{t+1}|s_t,a_t) * pi(a_t|s_t)`. + - When to use: as the definition every algorithm is trying to approximate. If an algorithm + optimizes something else (a Bellman residual, a surrogate objective), that substitution + is the source of its failure modes. + +- **The four value functions**: + - `V^pi(s)` — on-policy value: expected return starting in s and acting by pi forever. + - `Q^pi(s,a)` — on-policy action-value: same, but take an arbitrary a first. + - `V*(s)` — optimal value: expected return acting optimally from s. + - `Q*(s,a)` — optimal action-value: take arbitrary a, then act optimally. + - Two connections that come up constantly: `V^pi(s) = E_{a~pi}[Q^pi(s,a)]` and + `V*(s) = max_a Q*(s,a)`. + +- **The optimal-action shortcut**: if you have `Q*`, you get the optimal policy for free: + `a*(s) = argmax_a Q*(s,a)`. There may be several maximizers, all optimal, but there is + always an optimal policy that picks deterministically. + - When to use: this is the entire justification for Q-learning as a family — learn `Q*` + and the policy is a lookup. Ch17 explains what breaks when the argmax is over a + continuous space. + +- **Bellman equations**: "the value of your starting point is the reward you expect to get + from being there, plus the value of wherever you land next." + - On-policy: `V^pi(s) = E_{a~pi, s'~P}[r(s,a) + gamma V^pi(s')]` + - Optimal: `V*(s) = max_a E_{s'~P}[r(s,a) + gamma V*(s')]` + - The one crucial difference is the presence of the `max` over actions in the optimal form, + reflecting that an agent free to choose must pick the highest-value action. + - **Bellman backup** = the right-hand side, the reward-plus-next-value. + +- **The advantage function**: `A^pi(s,a) = Q^pi(s,a) - V^pi(s)`. How much better taking a + specific action is than randomly selecting one according to pi, assuming you follow pi + afterwards. + - When to use: whenever relative quality is what matters and absolute value is noise. + Crucially important to policy gradient methods (ch9). + +## Key Concepts +- **State vs observation**: a state s is a complete description of the world; an observation + o may omit information. Fully observed vs partially observed environments. Notation + routinely writes s where o is technically correct. +- **Action space**: the set of valid actions. **Discrete** (Atari, Go) versus **continuous** + (robot control, real-valued vectors). The distinction has profound consequences: some + algorithm families apply directly only to one case. +- **Policy**: deterministic `a_t = mu(s_t)` or stochastic `a_t ~ pi(.|s_t)`. In deep RL these + are **parameterized** — parameters theta or phi written as a subscript. "Policy" is often + used interchangeably with "agent." +- **Trajectory** (also **episode**, **rollout**): `tau = (s_0, a_0, s_1, a_1, ...)`. +- **Finite-horizon undiscounted return**: `R(tau) = sum_{t=0}^{T} r_t`. +- **Infinite-horizon discounted return**: `R(tau) = sum_{t=0}^{inf} gamma^t r_t`, with + `gamma` in (0,1). +- **MDP**: the 5-tuple ``. Markov property: transitions depend only on the + most recent state and action, not on prior history. + +## Code Examples +A deterministic continuous-action policy is just an MLP: + +```python +pi_net = nn.Sequential( + nn.Linear(obs_dim, 64), nn.Tanh(), + nn.Linear(64, 64), nn.Tanh(), + nn.Linear(64, act_dim) +) +obs_tensor = torch.as_tensor(obs, dtype=torch.float32) +actions = pi_net(obs_tensor) +``` + +**Categorical policies** (discrete actions) are built exactly like a classifier: observation +in, layers, a final linear layer giving logits per action, softmax to probabilities. Sampling +uses the framework's built-in categorical sampler. Log-likelihood is a vector index: +`log pi_theta(a|s) = log [P_theta(s)]_a`. + +**Diagonal Gaussian policies** (continuous actions) always have a network mapping observations +to mean actions `mu_theta(s)`. The covariance is diagonal, so it is a vector, represented one +of two ways: +1. A single **state-independent** vector of log standard deviations — standalone parameters. + *Spinning Up's VPG, TRPO and PPO do it this way.* +2. A network `log sigma_theta(s)` mapping states to log standard deviations, optionally + sharing layers with the mean network. *SAC does it this way, and SAC with state-independent + log stds did not work (ch19).* + +Log standard deviations, not standard deviations, because logs are free to range over +(-inf, inf) while stds must be nonnegative, and unconstrained parameters are easier to train. +Nothing is lost: exponentiate to recover. + +Sampling: `a = mu_theta(s) + sigma_theta(s) * z` with `z ~ N(0, I)` (elementwise product). + +Log-likelihood for a k-dimensional action: +`log pi_theta(a|s) = -0.5 * ( sum_i [ (a_i - mu_i)^2 / sigma_i^2 + 2 log sigma_i ] + k log 2pi )` + +## Mental Models +- **A discount factor is two arguments in one**: intuitively, cash now beats cash later; + mathematically, an infinite sum of rewards may not converge and is hard to work with. +- **The formalism is starker than the practice.** Deep RL routinely sets up algorithms to + optimize the *undiscounted* return while using discount factors when *estimating value + functions*. Expect the line to be blurred in real code. +- Treat **Q\* as a policy in disguise** and **V\* as a scoring function**: only the action-value + form directly yields an action. +- Ask of every value function: **is it time-dependent?** Unless stated otherwise, value + functions mean infinite-horizon discounted return. Finite-horizon undiscounted value + functions would need time as an argument — because what a state is worth depends on how + much time is left. + +## Anti-patterns +- **Reading `s` in an equation as a true state.** In partially observed settings the action + is conditioned on the observation; the notation is convention, not a claim. +- **Assuming an algorithm ports across action-space types.** Moving a method between discrete + and continuous spaces can require substantial rework, not a flag. +- **Learning `V*` and expecting to act.** Without `Q*` or a model you cannot extract the + action from the value. + +## Key Takeaways +1. The goal is always `argmax_pi E_{tau~pi}[R(tau)]`; everything else is machinery for + approximating it. +2. Four value functions, two of them optimal; `V^pi = E_a[Q^pi]` and `V* = max_a Q*` connect them. +3. Bellman equations hold for all four; the `max` is what separates optimal from on-policy. +4. `a*(s) = argmax_a Q*(s,a)` is why Q-learning is a viable family at all. +5. Advantage `A = Q - V` is the relative-quality signal that policy gradients are built on. +6. Log standard deviations are parameterized, not standard deviations, and state-independence + is an implementation choice that differs across the algorithms in this book. + +## Connects To +- **Ch 8**: the taxonomy built on what each algorithm chooses to learn. +- **Ch 9**: where the advantage function becomes the policy gradient weight. +- **Ch 17**: the continuous-action argmax problem, straight out of `a*(s) = argmax_a Q*(s,a)`. +- **Ch 12**: Exercise 1.1 asks you to implement the diagonal Gaussian log-likelihood above. diff --git a/engineering/spinning-up-deep-rl/skills/spinning-up-deep-rl/chapters/ch08-kinds-of-rl-algorithms.md b/engineering/spinning-up-deep-rl/skills/spinning-up-deep-rl/chapters/ch08-kinds-of-rl-algorithms.md new file mode 100644 index 00000000..11c5a9e6 --- /dev/null +++ b/engineering/spinning-up-deep-rl/skills/spinning-up-deep-rl/chapters/ch08-kinds-of-rl-algorithms.md @@ -0,0 +1,123 @@ +# Chapter 8: Part 2 — Kinds of RL Algorithms + +## Core Idea +Two branching questions organize modern RL: does the agent have or learn a **model** of the +environment, and **what does it learn** — a policy, a Q-function, a value function, or a +model. Every trade-off in algorithm design descends from those two choices. + +## Frameworks Introduced +- **Branch 1: model-free vs model-based.** A model is a function predicting state transitions + and rewards. + - **Upside of a model: it allows the agent to plan** — think ahead, see what would happen + across options, explicitly decide, then distil the planning results into a learned policy. + AlphaZero is the famous case. When it works, it substantially improves sample efficiency. + - **Downside: a ground-truth model is usually not available**, so it must be learned from + experience, and **bias in a learned model gets exploited by the agent** — producing an + agent that performs well against its own model and sub-optimally, or terribly, in the + real environment. Model-learning is fundamentally hard; large amounts of time and compute + can fail to pay off. + - When to use model-based: you have a reliable model, or sample efficiency dominates and + you can afford the model-bias risk. + +- **Branch 2: what to learn in model-free RL — two families.** + - **Policy Optimization.** Represent the policy explicitly as `pi_theta(a|s)` and optimize + theta either directly by gradient ascent on `J(pi_theta)` or indirectly by maximizing + local approximations of it. Almost always **on-policy**. Usually also learns an + approximator `V_phi(s)` used in figuring out the policy update. Examples: A2C/A3C + (direct), PPO (indirect, via a surrogate objective giving a conservative estimate of how + much `J` will change). + - **Q-Learning.** Learn an approximator `Q_theta(s,a)` to `Q*(s,a)`, usually with an + objective based on the Bellman equation. Almost always **off-policy**. The policy comes + from the connection `a(s) = argmax_a Q_theta(s,a)`. Examples: DQN, C51 (which learns a + distribution over return whose expectation is `Q*`). + +- **The policy-optimization vs Q-learning trade-off** (the single most quoted judgment in + this book): + - Policy optimization is **principled — you directly optimize for the thing you want** — + which tends to make it stable and reliable. + - Q-learning only *indirectly* optimizes agent performance, by training `Q_theta` to satisfy + a self-consistency equation. **There are many failure modes for this kind of learning, so + it tends to be less stable.** + - But when Q-learning works it is **substantially more sample efficient**, because it can + reuse data far more effectively. + - How to use: pick the failure you can detect and afford. Instability you can see in a + learning curve; sample inefficiency you can see in a compute bill. + +- **The interpolation principle**: policy optimization and Q-learning are not incompatible, + and under some circumstances turn out to be *equivalent* (Schulman et al 2017). A range of + algorithms lives between them and trades off deliberately: **DDPG** (learns a deterministic + policy and a Q-function that improve each other) and **SAC** (a variant using stochastic + policies, entropy regularization and other tricks to stabilize learning; scores higher than + DDPG on standard benchmarks). + +- **Four ways to use a model** (there is no small set of clean clusters here; the model may + be given or learned in each): + 1. **Pure planning** — never represent the policy at all. Model-predictive control (MPC): + each time the agent observes, compute a plan optimal with respect to the model over a + fixed window, execute only the first action, discard the rest, re-plan next step. Future + rewards past the horizon can enter through a learned value function. Example: MBMF. + 2. **Expert iteration** — keep an explicit policy `pi_theta`, use a planning algorithm + (e.g. Monte Carlo Tree Search) inside the model with candidate actions sampled from the + current policy. The planner's output is an "expert" relative to the policy; update the + policy toward it. Examples: ExIt, AlphaZero. + 3. **Data augmentation for model-free methods** — train a policy or Q-function with a + model-free algorithm, but augment real experience with fictitious experience (MBVE), or + train on purely fictitious experience ("training in the dream", World Models). + 4. **Embedding planning loops into policies** — make the planning procedure a subroutine of + the policy so complete plans become side information, and train the policy output with + any standard model-free algorithm. **The key advantage: model bias becomes less of a + problem, because where the model is bad for planning the policy can learn to ignore it.** + Example: I2A. + +## Key Concepts +- **Model of the environment**: a function predicting state transitions and rewards. +- **Surrogate objective**: a local approximation of `J(pi_theta)` that is safe to maximize; + the mechanism behind TRPO and PPO. +- **The deadly triad**: function approximation + bootstrapping + off-policy data, which + together cause instability in value-learning algorithms (Sutton and Barto ch. 11.3). This + is the concrete content of "Q-learning has many failure modes." +- **Modularity caveat**: the taxonomy is a tree and the real space is not. Advanced areas — + exploration, transfer learning, meta learning — are omitted from it entirely. + +## Reference Tables + +| Choice | Buys you | Costs you | +|--------|----------|-----------| +| Model-based | Planning; large sample-efficiency gains when the model is good | Model must usually be learned; model bias gets exploited | +| Model-free | Easier to implement and tune; more developed and tested | Forfeits the sample-efficiency gains a model could give | +| Policy optimization | Stability, reliability, directly optimizes performance | Sample inefficiency (on-policy data only) | +| Q-learning | Substantial sample efficiency through data reuse | Instability; no guarantee good Bellman fit means good policy | + +## Mental Models +- **Place any new algorithm with two questions**: model or no model, and what does it learn. + That is what the taxonomy is for; it does not need to be exhaustive to do that job. +- **Model bias is an adversarial problem, not a noise problem.** The agent is actively + optimizing against your model's errors. That is why "the model is only 95% accurate" does + not translate into "the policy is 95% as good." +- **"Directly optimize the thing you want" is a design principle you can apply outside RL**: + the further your training objective sits from your evaluation metric, the more failure modes + you inherit. + +## Anti-patterns +- **Reading the taxonomy as exhaustive or as a strict tree.** The book opens with that + disclaimer: algorithm modularity is not well represented by a tree. +- **Assuming model-based is strictly better because it plans.** As of the guide's writing + (September 2018), model-free methods were more popular and more extensively developed and + tested, precisely because model-learning is hard. +- **Treating "it fits the Bellman equation well" as evidence the policy is good.** No such + guarantee exists — this is the defining weakness of the whole Q-learning family. + +## Key Takeaways +1. Two branching questions — model or not, and what to learn — generate the whole landscape. +2. Policy optimization: principled, stable, sample-hungry. Q-learning: sample-efficient, + indirect, unstable. +3. The two families are not exclusive; DDPG and SAC live between them on purpose. +4. Model bias is exploited by the agent, which is the central risk of model-based methods. +5. Embedding a planner inside a policy is the model-based approach that most directly + defuses model bias, because the policy can learn when to ignore the model. + +## Connects To +- **Ch 3**: the six implemented algorithms, which are all model-free. +- **Ch 9**: the mathematics of the policy-optimization family. +- **Ch 17 and Ch 19**: DDPG and SAC as the concrete interpolations named here. +- **Ch 11**: the key-papers list, whose top-level sections mirror this taxonomy. diff --git a/engineering/spinning-up-deep-rl/skills/spinning-up-deep-rl/chapters/ch09-intro-to-policy-optimization.md b/engineering/spinning-up-deep-rl/skills/spinning-up-deep-rl/chapters/ch09-intro-to-policy-optimization.md new file mode 100644 index 00000000..5201ee4b --- /dev/null +++ b/engineering/spinning-up-deep-rl/skills/spinning-up-deep-rl/chapters/ch09-intro-to-policy-optimization.md @@ -0,0 +1,161 @@ +# Chapter 9: Part 3 — Intro to Policy Optimization + +## Core Idea +Three results build the policy gradient you actually implement: the simplest analytical +expression for `grad J`, a rule that lets you **drop useless terms** (reward-to-go), and a +rule that lets you **add useful terms** (baselines) — ending at the advantage-weighted form +`grad J = E[ sum_t grad log pi_theta(a_t|s_t) * A^pi(s_t,a_t) ]`. + +## Frameworks Introduced +- **The general policy gradient form**: every variant is + `grad_theta J(pi_theta) = E_{tau~pi_theta}[ sum_{t=0}^{T} grad_theta log pi_theta(a_t|s_t) * Phi_t ]` + and the whole subject is *which* `Phi_t` you choose. Five valid choices, all with the same + expectation and different variance: + 1. `Phi_t = R(tau)` — the full trajectory return (the simplest form) + 2. `Phi_t = sum_{t'=t}^{T} R(s_t', a_t', s_{t'+1})` — the **reward-to-go** + 3. `Phi_t = reward-to-go - b(s_t)` — reward-to-go with a **baseline** + 4. `Phi_t = Q^{pi_theta}(s_t, a_t)` + 5. `Phi_t = A^{pi_theta}(s_t, a_t)` — the **advantage**, the one Spinning Up's VPG uses + - When to use: choice 5 in practice; the list is the map for reading any policy gradient paper. + +- **The derivation, in five facts.** Worth memorizing because every extension reuses them: + 1. `P(tau|theta) = rho_0(s_0) * prod_t P(s_{t+1}|s_t,a_t) * pi_theta(a_t|s_t)` + 2. **Log-derivative trick**: `grad_theta P(tau|theta) = P(tau|theta) * grad_theta log P(tau|theta)` + 3. `log P(tau|theta) = log rho_0(s_0) + sum_t [ log P(s_{t+1}|s_t,a_t) + log pi_theta(a_t|s_t) ]` + 4. **Gradients of environment functions are zero** — the environment has no dependence on theta, + so `rho_0`, `P` and `R` all vanish under `grad_theta`. + 5. Therefore `grad_theta log P(tau|theta) = sum_t grad_theta log pi_theta(a_t|s_t)`. + - Chained: expand the expectation, bring the gradient inside the integral, apply the + log-derivative trick, return to expectation form, substitute fact 5. + +- **The EGLP lemma (Expected Grad-Log-Prob)**: for any parameterized distribution `P_theta`, + `E_{x~P_theta}[ grad_theta log P_theta(x) ] = 0`. + - Proof in three lines: all distributions are normalized (`integral P_theta(x) = 1`); take + the gradient of both sides (`= grad 1 = 0`); apply the log-derivative trick. + - When to use: it is the engine behind both the drop rule and the add rule. The author notes + it has no standard name in the literature but comes up often enough to deserve one. + +- **"Don't let the past distract you" (the drop rule)**: agents should only reinforce actions + on the basis of their **consequences**. Rewards obtained *before* an action have no bearing + on how good that action was. Formally, all terms with `t' < t` are zero in expectation. + - Why it is better, precisely: those dropped terms had **zero mean but nonzero variance** — + they added pure noise to the sample estimate. Removing them reduces the number of sample + trajectories needed. + +- **Baselines (the add rule)**: an immediate consequence of EGLP is that for any function `b` + depending only on state, `E_{a_t~pi}[ grad log pi(a_t|s_t) * b(s_t) ] = 0`. So you may add + or subtract any such term without changing the gradient in expectation. + - **The most common baseline is the on-policy value function `V^pi(s_t)`**, which empirically + reduces variance and gives faster, more stable learning. + - The conceptual appeal: it encodes the intuition that **if an agent gets what it expected, + it should "feel" neutral about it.** + - In practice `V^pi` cannot be computed exactly, so it is approximated by a network `V_phi` + updated concurrently with the policy (so it always approximates the *most recent* policy). + The simplest learning rule, used by VPG, TRPO, PPO and A2C, is mean-squared error: + `phi_k = argmin_phi E_{s_t, Rhat_t ~ pi_k}[ (V_phi(s_t) - Rhat_t)^2 ]`, via one or more + gradient steps starting from `phi_{k-1}`. + +## Key Concepts +- **Policy gradient**: `grad_theta J(pi_theta)`. Algorithms that optimize this way are + policy gradient algorithms — VPG and TRPO are; **PPO is often called one though this is + slightly inaccurate.** +- **Reward-to-go**: `Rhat_t = sum_{t'=t}^{T} R(s_t', a_t', s_{t'+1})`. +- **Baseline**: any state-only function subtracted from the weight. +- **Sample estimate**: `ghat = (1/|D|) * sum_{tau in D} sum_t grad log pi_theta(a_t|s_t) * Phi_t`. +- **GAE (Generalized Advantage Estimation)**: the widely-used method for approximating the + advantage function; Spinning Up's VPG, TRPO and PPO all use it. The book strongly advises + studying the paper. +- **Epoch** (in this context): one experience-collection phase plus one policy gradient update. + +## Code Examples +The whole simple algorithm is 128 lines (`spinup/examples/pytorch/pg_math/1_simple_pg.py`). +The three pieces that matter: + +```python +# 1. Policy network +logits_net = mlp(sizes=[obs_dim] + hidden_sizes + [n_acts]) + +def get_policy(obs): + return Categorical(logits=logits_net(obs)) + +def get_action(obs): + return get_policy(obs).sample().item() + +# 2. The "loss" whose gradient is the policy gradient +def compute_loss(obs, act, weights): + logp = get_policy(obs).log_prob(act) + return -(logp * weights).mean() + +# 3. One gradient step +optimizer.zero_grad() +batch_loss = compute_loss(obs=..., act=..., weights=...) +batch_loss.backward() +optimizer.step() +``` + +Upgrading to reward-to-go changes one function and two lines: + +```python +def reward_to_go(rews): + n = len(rews) + rtgs = np.zeros_like(rews) + for i in reversed(range(n)): + rtgs[i] = rews[i] + (rtgs[i+1] if i+1 < n else 0) + return rtgs + +# was: batch_weights += [ep_ret] * ep_len +batch_weights += list(reward_to_go(ep_rews)) +``` + +For a diagonal Gaussian policy, `log_prob(act)` returns per-component log probabilities of +shape `(batch, act_dim)` when RL needs shape `(batch,)`. Sum them: + +```python +logp = get_policy(obs).log_prob(act).sum(axis=-1) +``` + +## Worked Example +**The policy gradient "loss" is not a loss function.** This is the single most consequential +warning in the chapter, and it differs from supervised learning in two ways: + +1. **The data distribution depends on the parameters.** A supervised loss is defined on a + fixed data distribution independent of the parameters being optimized. Here the data must + be sampled from the most recent policy. +2. **It does not measure performance.** We care about `J(pi_theta)`, and this "loss" does not + approximate it — not even in expectation. It is useful only because, *evaluated at the + current parameters with data generated by those parameters*, it has the negative gradient + of performance. + +After the first gradient step there is no connection to performance at all. You can send +this loss to negative infinity while policy performance craters — and it usually will. +Researchers sometimes call this the policy "overfitting" to a batch; the phrase is +descriptive but should not be taken literally, since it does not refer to generalization error. + +**The practical rule: in policy gradients, only average return means anything. The loss +function means nothing.** The ML habit of reading "loss went down, all is well" is wrong here. + +## Anti-patterns +- **Using the loss curve as a training health signal.** See above. Watch `AverageEpRet`. +- **Weighting every action by the full-trajectory return** when reward-to-go is a two-line + change that strictly reduces variance. +- **Forgetting to sum log-probabilities across action dimensions** for Gaussian policies — + a shape bug that will not raise, in the same family as the ch12 DDPG bug. +- **Treating a baseline as a bias**: baselines are provably zero-mean under EGLP. They change + variance, never the expected gradient. + +## Key Takeaways +1. One template, five valid weights `Phi_t`; the advantage form is the destination. +2. The whole derivation rests on the log-derivative trick plus the fact that environment + gradients vanish. +3. EGLP is a single lemma that licenses both dropping past rewards and adding baselines. +4. Reward-to-go removes terms with zero mean and nonzero variance — pure noise reduction. +5. The value baseline is learned by mean-squared regression onto reward-to-go, concurrently + with the policy. +6. The policy gradient loss is not a performance measure and must never be read as one. + +## Connects To +- **Ch 7**: the advantage function and value functions this chapter puts to work. +- **Ch 14**: VPG, the direct implementation of the final advantage-weighted form. +- **Ch 15 and Ch 16**: TRPO and PPO, which replace the plain gradient step with a + trust-region-constrained one. +- **Ch 12**: Exercise 2.1 measures exactly what happens when `V_phi` is not fit properly. diff --git a/engineering/spinning-up-deep-rl/skills/spinning-up-deep-rl/chapters/ch10-spinning-up-as-a-researcher.md b/engineering/spinning-up-deep-rl/skills/spinning-up-deep-rl/chapters/ch10-spinning-up-as-a-researcher.md new file mode 100644 index 00000000..b248b22d --- /dev/null +++ b/engineering/spinning-up-deep-rl/skills/spinning-up-deep-rl/chapters/ch10-spinning-up-as-a-researcher.md @@ -0,0 +1,133 @@ +# Chapter 10: Spinning Up as a Deep RL Researcher + +## Core Idea +Joshua Achiam's curriculum for becoming a deep RL researcher, in four stages: build the +right background, learn by writing your own implementations, develop a research project +through one of three idea frames, and hold yourself to experimental standards strict enough +that a positive result means something. + +## Frameworks Introduced +- **The Right Background** (four items, all prerequisites, none optional): + - **Math**: from probability and statistics — random variables, Bayes' theorem, chain rule + of probability, expected values, standard deviations, importance sampling. From + multivariate calculus — gradients, and optionally Taylor series expansions. + - **General deep learning**: standard architectures (MLP, vanilla RNN, LSTM, GRU, conv + layers, resnets, attention), regularizers (weight decay, dropout), normalization (batch, + layer, weight norm), optimizers (SGD, momentum SGD, Adam), and the reparameterization trick. + - **One deep learning library**, well enough to confidently implement a simple supervised + learning program. + - **RL concepts and terminology**: states, actions, trajectories, policies, rewards, value + functions, action-value functions. Optionally, monotonic improvement theory (the basis + for advanced policy gradient algorithms) and classical RL algorithms. + +- **Learn by Doing** — the core of the essay, in eight rules: + 1. **Write your own implementations**, aiming for the **shortest correct implementation** of + each. By far the best way to build both understanding and performance intuitions. + 2. **Simplicity is critical.** Implement the simplest algorithms first and add complexity + gradually. Starting with too many moving parts means weeks lost to debugging — a common + failure mode for newcomers. If you are stuck in it, drop to a simpler algorithm and + return later. + 3. **Which algorithms, in roughly this order**: VPG (REINFORCE), DQN, A2C (the synchronous + A3C), PPO (clipped objective), DDPG. The simplest versions are a few hundred lines + (ballpark 250-300); a no-frills VPG is about 80. Write single-threaded before parallel, + but do parallelize at least one. + 4. **Focus on understanding, because broken RL code almost always fails silently** — the + code runs fine and the agent simply never learns. Usually something is computed with the + wrong equation, on the wrong distribution, or piped to the wrong place. Sometimes the + only way to find it is to read the code critically, knowing exactly what it should do. + 5. **What to look for in papers**: scour ablation analyses (they tell you which parameters + and subroutines actually matter, which is what you need to diagnose bugs) and + supplementary material (network architectures, optimization hyperparameters — align your + implementation to these to improve your odds of getting it working). + 6. **But do not overfit to paper details.** Papers often prescribe more tricks than are + strictly necessary. The original DDPG paper suggests a complex architecture, an + initialization scheme and batch normalization; none are strictly necessary and some of + the best DDPG results use simpler networks. The original A3C uses asynchronous updates + from actor-learners; synchronous updates work about as well. + 7. **Do not overfit to existing implementations either.** RL libraries make abstraction + choices that are good for code reuse across algorithms but unnecessary if you are + writing one algorithm for one use case. + 8. **Iterate fast in simple environments.** CartPole-v0, InvertedPendulum-v0, FrozenLake-v0, + HalfCheetah-v2 with a short horizon (100 or 250 steps, not the full 1000). + **Ideal debug-stage turnaround: under 5 minutes on your local machine.** These runs need + no special hardware and run fine on CPUs. Do not attempt Atari or Humanoid before the + simplest toy task works. + +- **Three idea-generation frames** — the frame you pick shapes the project's scope and risk: + + | Frame | What it is | Scope | Main risk | + |-------|-----------|-------|-----------| + | **1. Improving on an existing approach** | Incrementalism: performance gains in an established setting by tweaking an existing algorithm. Reimplementing prior work exposes where it is brittle. Most accessible to novices, still worthwhile at any level | Narrow, wraps up in a few months | Your tweaks may simply fail to improve it, and then the project is over with no signal on what to do next | + | **2. Focusing on unsolved benchmarks** | Succeed at a task nobody has solved. You may try a wide variety of methods, prior and invented | Broad, several months to a year-plus | The benchmark may be unsolvable without a substantial breakthrough; easy to spend a long time with no progress. Even failure usually yields insights that seed the next project | + | **3. Creating a new problem setting** | An entirely different conceptual problem nobody has studied; you will have to design the benchmark too | Open-ended | Enormous challenge — but great benchmarks move the whole field forward. These problems come up when they come up; hard to go looking for them | + +- **Doing Rigorous Research in RL** — four standards, each closing a specific way a result + can look real and not be: + 1. **Set up fair comparisons.** If you implement your baseline from scratch, spend **as much + time tuning the baseline as tuning your own algorithm.** Hold all else equal even when + the methods differ substantially — e.g. keep parameter counts approximately equal when + investigating architecture variants. **Under no circumstances handicap the baseline.** + RL baselines are strong, and consistent wins over them are hard. + 2. **Remove stochasticity as a confounder.** Run everything for many random seeds — **at + least 3, and 10 or more to be thorough.** Deep RL is fairly brittle with respect to seed; + two different groups of seeds can produce learning curves so different they look like + they come from different distributions. + 3. **Run high-integrity experiments.** Do not report the best or most interesting runs. + Launch **new, final experiments** for every method being compared and **precommit to + reporting whatever comes out.** This enforces a weak form of preregistration: the tuning + stage produces your hypotheses, the final runs produce your conclusions. + 4. **Check each claim separately — run an ablation analysis.** Any proposed method has + several key design decisions, and the claim "these collectively help" is really a bundle + of separate claims. Systematically swap or remove each one to attribute credit correctly. + This lets you state each claim with a measure of confidence and strengthens the whole work. + +## Key Concepts +- **Silent failure**: the defining property of broken RL code. It runs; the agent just never learns. +- **Measure everything**: instrument heavily. The author's own list — mean/std/min/max of + cumulative rewards, episode lengths and value function estimates, plus the objective losses + and any exploration parameters (mean policy entropy for stochastic policy optimization, + current epsilon for epsilon-greedy). **Also watch videos of your agent** periodically; it + gives insights nothing else does. You cannot tell it is broken if you cannot see it breaking. +- **Assume there is a bug.** Spend a lot of effort searching for bugs before tweaking + hyperparameters. Bad hyperparameters can significantly degrade performance, but if yours + are similar to those in papers and standard implementations, they are probably not the issue. +- **Test in more than one environment**: sometimes code works in one environment despite a + breaking bug, so re-verify once results look promising. +- **Avoid reinventing the wheel**: before investing, check thoroughly that the idea has not + been done. But do not let the risk push you into planting flags with not-quite-finished + research or over-claiming partial work. Complete, thorough investigations are what counts. +- **Scale when things work**: after the simplest environments pass, move to harder ones — + experiments now take hours to a couple of days, and specialized hardware (a beefy GPU, a + 32-core machine) or cloud resources start to be worth it. + +## Anti-patterns +- **Tuning hyperparameters to fix a bug.** It is usually a bug. +- **Debugging in a hard environment.** If turnaround exceeds a few minutes at the debug stage, + you are working in the wrong environment. +- **Starting with the complex algorithm.** The predicted outcome is weeks lost. +- **Reporting your best run.** That is the failure the precommitment rule exists to stop. +- **Under-tuning the baseline** — the most common way to manufacture a positive result without + intending to. +- **Fewer than 3 seeds.** With deep RL's seed sensitivity, a single-seed result carries + almost no information. +- **Bundling claims.** Without ablations you cannot say which of your design decisions did + the work — including, possibly, none of them. + +## Key Takeaways +1. Implement from scratch, simplest first, shortest correct version, single-threaded before parallel. +2. Broken RL code fails silently — so instrumentation and critical reading, not error messages, + are your debugging tools. +3. Debug loops under five minutes in toy environments; scale only after correctness. +4. Read papers for ablations and supplementary material, but do not adopt every trick. +5. Pick an idea frame deliberately: incremental (narrow, fast, may dead-end), unsolved + benchmark (broad, slow, insight-rich even in failure), or new problem setting (rare, huge). +6. Rigor is four things: a fully-tuned baseline, many seeds, precommitted final runs, and + per-claim ablations. +7. These habits are worth keeping past the learning stage; they accelerate research. + +## Connects To +- **Ch 1**: the code design philosophy that makes the reference implementations readable. +- **Ch 4**: the `--seed 0 10 20` flag that operationalizes the multi-seed rule. +- **Ch 11**: the key papers list, the recommended starting point for literature exploration. +- **Ch 12**: the exercises — Problem Set 2 is entirely about silent failure modes. +- **Ch 13**: the benchmarks, run at 10 seeds — the standard this chapter asks for. diff --git a/engineering/spinning-up-deep-rl/skills/spinning-up-deep-rl/chapters/ch11-key-papers-in-deep-rl.md b/engineering/spinning-up-deep-rl/skills/spinning-up-deep-rl/chapters/ch11-key-papers-in-deep-rl.md new file mode 100644 index 00000000..f65afdba --- /dev/null +++ b/engineering/spinning-up-deep-rl/skills/spinning-up-deep-rl/chapters/ch11-key-papers-in-deep-rl.md @@ -0,0 +1,70 @@ +# Chapter 11: Key Papers in Deep RL + +## Core Idea +A curated, explicitly non-comprehensive reading list of roughly 100 papers organized into +13 topic areas — designed as a starting point for someone looking to do research, and as +the map you use to pick a research topic in ch10's "explore the literature" step. + +## Frameworks Introduced +- **Use the list as a topic map, not a queue.** The intended workflow (from ch10): scan the + sections to become aware of what topics exist, find a paper on one that inspires you, read + it thoroughly, then use its related-work section and citations to do a deep dive. You will + start to see where the unsolved problems are. +- **Each entry is annotated with what it contributes** — either **Algorithm: X** (this paper + introduced X) or **Contribution: ...** (this paper established a result, a critique or a + codebase). Scan the annotations, not the titles. + +## Reference Tables + +The 13 sections, with the subsections that reveal the field's internal structure: + +| # | Section | Subsections / notable entries | +|---|---------|------------------------------| +| 1 | **Model-Free RL** | a. Deep Q-Learning (DQN, Deep Recurrent Q-Learning, Dueling DQN, Double DQN, Prioritized Experience Replay, Rainbow) · b. Policy Gradients (A3C, TRPO, GAE, PPO, ACKTR, ACER, SAC) · c. Deterministic Policy Gradients (DPG, DDPG, TD3) · d. Distributional RL (C51, QR-DQN, IQN, Dopamine) · e. Policy Gradients with Action-Dependent Baselines (Q-Prop, Stein Control Variates, and Tucker et al's critique) · f. Path-Consistency Learning (PCL, Trust-PCL) · g. Other Ways of Combining Policy-Learning and Q-Learning (PGQL, Reactor, IPG, the policy-gradient/soft-Q-learning equivalence) · h. Evolutionary Algorithms (ES) | +| 2 | **Exploration** | a. Intrinsic Motivation (VIME, count-based pseudocounts, hash-based counts, EX2, ICM, RND) · b. Unsupervised RL (VIC, DIAYN, VALOR) | +| 3 | **Transfer and Multitask RL** | Progressive Networks, UVFA, UNREAL, PathNet, MATL, HER | +| 4 | **Hierarchy** | STRAW, Feudal Networks, HIRO | +| 5 | **Memory** | MFEC, NEC, Neural Map, MERLIN, RMC | +| 6 | **Model-Based RL** | a. Model is Learned (I2A, MBMF, MVE) · b. Model is Given (AlphaZero and relatives) | +| 7 | **Meta-RL** | | +| 8 | **Scaling RL** | | +| 9 | **RL in the Real World** | | +| 10 | **Safety** | | +| 11 | **Imitation Learning and Inverse Reinforcement Learning** | | +| 12 | **Reproducibility, Analysis, and Critique** | | +| 13 | **Bonus: Classic Papers in RL Theory or Review** | | + +## Mental Models +- **Section 1's shape mirrors ch8's taxonomy**: Q-learning, policy gradients, and the + deterministic and interpolating methods between them. If you understood ch8, section 1 + is already organized in your head. +- **Section 12 is the one people skip and shouldn't.** "Reproducibility, Analysis, and + Critique" is where ch10's rigor standards come from, and where you learn that published + gains sometimes do not survive re-examination — the Tucker et al entry in section 1e is a + worked example: it critiques and re-evaluates claims from earlier papers (including Q-Prop + and Stein control variates) and finds important methodological errors in them. +- **A reading list is a topic-awareness tool.** The list explicitly does not claim + completeness; its job is to prevent you from picking a research problem while unaware that + a whole subfield already works on it. + +## Anti-patterns +- **Reading it front to back.** It is far from comprehensive and is not a curriculum; the + intended use is targeted depth after topic selection. +- **Treating an entry as endorsement of the result.** Several entries exist precisely because + they critique other entries. +- **Skipping the related-work walk.** The list is the entry point; the citations are the + actual literature review. + +## Key Takeaways +1. 13 topic sections; section 1 (Model-Free RL) alone has eight subsections and mirrors ch8. +2. Every entry is annotated with the algorithm it introduced or the contribution it made. +3. The list is a topic map for choosing a research direction, not a reading queue. +4. Sections 7-13 (meta-RL, scaling, real world, safety, imitation/IRL, critique, classics) + are where the topics ch10 suggests for project ideas actually live. + +## Connects To +- **Ch 8**: the taxonomy that section 1's structure reproduces. +- **Ch 10**: "start by exploring the literature to become aware of topics in the field" — + this chapter is that step's tool. +- **Ch 14-19**: every implemented algorithm's own "Why These Papers?" section is a + three-paper version of this list, scoped to one algorithm. diff --git a/engineering/spinning-up-deep-rl/skills/spinning-up-deep-rl/chapters/ch12-exercises.md b/engineering/spinning-up-deep-rl/skills/spinning-up-deep-rl/chapters/ch12-exercises.md new file mode 100644 index 00000000..57e42e3b --- /dev/null +++ b/engineering/spinning-up-deep-rl/skills/spinning-up-deep-rl/chapters/ch12-exercises.md @@ -0,0 +1,105 @@ +# Chapter 12: Exercises + +## Core Idea +Two problem sets with opposite purposes: Problem Set 1 makes you write the mathematical +core of three algorithms, and Problem Set 2 makes you **watch RL code fail silently** and +find out why — because that is the skill the essay says decides whether you can do this work. + +## Frameworks Introduced +- **Problem Set 1: Basics of Implementation** — you are given everything except the math. + - **1.1 Gaussian Log-Likelihood.** Write a function taking means, log stds and samples, + returning the log likelihoods. Auto-checked against a known-good implementation on a + batch of random inputs. The formula is the one in ch7. + - **1.2 Policy for PPO.** Implement an MLP diagonal Gaussian policy. Auto-checked by + running 20 epochs on InvertedPendulum-v2 (3-5 minutes). **Bar for success: average score + over 500 in the last 5 epochs, or 1000 (the maximum) in the last 5.** + - **1.3 Computation Graph for TD3.** You are given the entire TD3 algorithm *except* the + loss functions and the intermediate calculations for them. Find "YOUR CODE HERE". + No automatic checking. Evaluated on HalfCheetah-v2, InvertedPendulum-v2 and one other + MuJoCo environment of your choosing, with smaller networks ([128,128]), max episode + length 150, 10 epochs, roughly 10 minutes. **Anecdotal targets within 10 epochs: + HalfCheetah over 300, InvertedPendulum maxing out at 150.** `--use_soln` runs Spinning + Up's TD3 instead of yours. + +- **Problem Set 2: Algorithm Failure Modes** — the point is the failure, not the fix. + - **2.1 Value Function Fitting in TRPO.** Compare `train_v_iters=80` against + `train_v_iters=0` on Hopper-v2, three seeds each, 250 epochs, 4000 steps per epoch. + **Result: the difference is substantial. With a trained value function the agent makes + quick progress; with an untrained one it gets stuck early on.** Few factors affect policy + gradient performance more drastically than the quality of the value function used for + advantage estimation. + - **2.2 Silent Bug in DDPG.** Run DDPG with and without a planted bug, three seeds each, + six runs, ~10 minutes each; plot and compare. Then, **without looking at DDPG's `core.py`**, + work out what the bug is. + +## Worked Example +**The 2.2 bug, and why it is the most instructive page in the book.** + +The correct and bugged actor-critic differ in exactly one thing: whether the Q-function +output is squeezed. + +```python +# Correct +def forward(self, obs, act): + q = self.q(torch.cat([obs, act], dim=-1)) + return torch.squeeze(q, -1) # Critical to ensure q has right shape. + +# Bugged +def forward(self, obs, act): + return self.q(torch.cat([obs, act], dim=-1)) # shape [batch, 1], not [batch] +``` + +The TF1 version is the same defect: the correct code squeezes to shape `[batch size]`, the +bugged code leaves shape `[batch size, 1]`. + +Why that one missing squeeze destroys learning — look at the DDPG graph: + +```python +backup = tf.stop_gradient(r_ph + gamma*(1-d_ph)*q_pi_targ) +pi_loss = -tf.reduce_mean(q_pi) +q_loss = tf.reduce_mean((q - backup)**2) +``` + +`r_ph` and `d_ph` have shape `[batch size]`. The backup line was written assuming it adds +tensors of the same shape. But it will also happily add tensors of *different* shapes as long +as they are broadcast-compatible — and `[batch size]` and `[batch size, 1]` are compatible, +with results that are not what you expect: + +``` +x has shape [5], y has shape [5,1] +x * y -> shape [5,5] +x + y -> shape [5,5] +``` + +Adding or multiplying a shape-`[5]` tensor by a shape-`[5,1]` tensor returns a shape-`[5,5]` +tensor. So when the Q-functions are not squeezed, `q_pi_targ` has shape `[batch size, 1]`, +the backup becomes a `[batch, batch]` matrix, and the whole Q-loss is meaningless. **Nothing +raises. Nothing warns. The run completes. The agent just learns worse.** + +This is ch10's "broken RL code almost always fails silently" reduced to a single missing +`squeeze`. The **Bonus** question is worth sitting with: *are there any choices of +hyperparameters which would have hidden the effects of the bug?* + +## Anti-patterns +- **Looking at `core.py` before attempting 2.2.** The exercise is diagnostic practice; the + answer is worth less than the search. +- **Broadcast-compatible shapes as an implicit contract.** The lesson generalizes far past + DDPG: any elementwise op between a `[N]` and an `[N,1]` tensor is a silent bug waiting. + Assert shapes. +- **Skipping Problem Set 2 because it produces no code.** 2.1 produces no code at all and is + still one of the most decision-relevant results in the book. + +## Key Takeaways +1. Problem Set 1 is math-in-code: Gaussian log-likelihood, a PPO policy, TD3's losses. +2. 1.1 and 1.2 self-check; 1.3 does not, and is evaluated on visible learning progress. +3. 2.1: a badly-fit value function does not degrade a policy gradient agent gracefully — it + gets stuck early. +4. 2.2: one missing `squeeze` silently corrupts the Bellman backup through broadcasting. +5. The challenges past the problem sets are: reimplement algorithms from scratch (ch10), and + attempt OpenAI's standing Requests for Research. + +## Connects To +- **Ch 7**: the diagonal Gaussian log-likelihood formula that Exercise 1.1 asks for. +- **Ch 9**: why the value function baseline quality drives policy gradient performance (2.1). +- **Ch 17 and Ch 18**: the DDPG graph and the TD3 losses the exercises operate on. +- **Ch 10**: "broken RL code almost always fails silently" — Problem Set 2 is the proof. diff --git a/engineering/spinning-up-deep-rl/skills/spinning-up-deep-rl/chapters/ch13-benchmarks.md b/engineering/spinning-up-deep-rl/skills/spinning-up-deep-rl/chapters/ch13-benchmarks.md new file mode 100644 index 00000000..38ed29f7 --- /dev/null +++ b/engineering/spinning-up-deep-rl/skills/spinning-up-deep-rl/chapters/ch13-benchmarks.md @@ -0,0 +1,87 @@ +# Chapter 13: Benchmarks for Spinning Up Implementations + +## Core Idea +All six algorithms were benchmarked for 3M timesteps across five MuJoCo environments at +10 seeds each — and the results carry an explicit, unusual disclosure about which +implementations are research-grade and which are not. + +## Frameworks Introduced +- **The parity disclosure — the single most decision-relevant paragraph in the chapter:** + - **DDPG, TD3 and SAC are roughly at parity with the best reported results** for those + algorithms. **You can use them for research purposes.** + - **VPG, TRPO and PPO are overall a bit weaker than the best reported results**, because + the implementations omit standard tricks — specifically **observation normalization** and + **normalized value regression targets**. **For research comparisons, use the TRPO or PPO + implementations from OpenAI Baselines instead.** + - When to use: before citing any Spinning Up number, or before using one of these as a + baseline in a paper (which ch10 says you must tune as hard as your own method). + +- **Report performance with its measurement definition attached.** The two families are not + measured the same way, and saying "return" without saying which is a category error: + - **On-policy**: the average trajectory return across the batch collected at each epoch — + i.e. the exploring policy, measured continuously. + - **Off-policy**: measured **once every 10,000 steps** by running the deterministic policy + (or, for SAC, the **mean** policy) **without action noise** for **ten trajectories**, and + reporting the average return over those test trajectories. + +## Reference Tables + +Benchmark setup: + +| Parameter | On-policy (VPG, TRPO, PPO) | Off-policy (DDPG, TD3, SAC) | +|-----------|---------------------------|----------------------------| +| Network architecture | (64, 32), tanh units | (256, 256), relu units | +| Batch | 4000 environment steps per batch update | Minibatches of 100 per gradient step | +| Performance metric | Average batch trajectory return per epoch | Deterministic/mean policy, no noise, 10 trajectories every 10k steps | +| Research-grade? | No — use Baselines for TRPO/PPO | Yes | + +Common to all: + +| Setting | Value | +|---------|-------| +| Environments | HalfCheetah-v3, Hopper-v3, Walker2d-v3, Swimmer-v3, Ant-v3 (MuJoCo Gym suite) | +| Timesteps | 3M | +| Random seeds | 10 per experiment | +| Plot content | Solid line = mean over seeds; shaded = std dev over seeds | +| Smoothing | Averaged over a window of 11 epochs | +| Other hyperparameters | Left at Spinning Up defaults; see each algorithm page | + +Both PyTorch and TF1 versions were benchmarked in every environment, plus dedicated +head-to-head PyTorch-vs-TF1 pages for VPG, PPO, DDPG, TD3 and SAC. (TRPO has no PyTorch +implementation, so it has no head-to-head page.) + +## Mental Models +- **Reporting mean and std dev over 10 seeds *as shaded bands* is the visual form of ch10's + "remove stochasticity as a confounder."** A benchmark plot without a seed band is telling + you less than it appears to. +- **The architecture split is itself a finding**: on-policy at (64, 32)/tanh and off-policy + at (256, 256)/relu is the convention these algorithm families settled into, not an + arbitrary choice, and copying the wrong family's defaults is a real source of bad results. +- **Honest self-assessment is a feature of the resource**, not a caveat. A benchmarks page + that names which of its own implementations you should not use is doing the reader's + ch10 rigor work for them. + +## Anti-patterns +- **Using Spinning Up's PPO or TRPO as a paper baseline.** The page says explicitly to use + Baselines for those. Under-powered baselines are the ch10 failure of "handicapping the + baseline," even when it is unintentional. +- **Comparing an on-policy `AverageEpRet` against an off-policy `AverageTestEpRet`** as if + they were the same quantity. Use the plotter's `Performance` alias (ch6), which resolves + per family. +- **Reading a single-seed curve as a result.** These plots are 10 seeds precisely because + fewer is not informative. + +## Key Takeaways +1. 3M timesteps, five MuJoCo environments, 10 seeds, both backends. +2. DDPG, TD3, SAC: research-usable. VPG, TRPO, PPO: educational only — use Baselines. +3. The weakness is attributed to two named omissions: observation normalization and + normalized value regression targets. +4. On-policy and off-policy performance are different measurements, not different numbers + of the same measurement. +5. Architecture defaults differ by family: (64,32)/tanh on-policy, (256,256)/relu off-policy. + +## Connects To +- **Ch 1**: "reasonably good" — this chapter is the quantification of that phrase. +- **Ch 6**: the `Performance` alias, which exists because of the metric split described here. +- **Ch 10**: the seed and fair-baseline standards this benchmark actually meets. +- **Ch 14-19**: per-algorithm hyperparameter defaults referenced as "left at default settings." diff --git a/engineering/spinning-up-deep-rl/skills/spinning-up-deep-rl/chapters/ch14-vpg.md b/engineering/spinning-up-deep-rl/skills/spinning-up-deep-rl/chapters/ch14-vpg.md new file mode 100644 index 00000000..feb97d5b --- /dev/null +++ b/engineering/spinning-up-deep-rl/skills/spinning-up-deep-rl/chapters/ch14-vpg.md @@ -0,0 +1,93 @@ +# Chapter 14: Vanilla Policy Gradient (VPG) + +## Core Idea +Push up the probabilities of actions that lead to higher return and push down the +probabilities of actions that lead to lower return, until you arrive at the optimal policy. +That is the whole idea underlying policy gradients, and VPG is its most direct expression. + +## Quick Facts +- **On-policy.** +- Works with **discrete or continuous** action spaces. +- The Spinning Up implementation **supports MPI parallelization**. + +## Key Equations +The advantage-weighted policy gradient (the end point of ch9): + +``` +grad_theta J(pi_theta) = E_{tau ~ pi_theta}[ sum_{t=0}^{T} grad_theta log pi_theta(a_t|s_t) * A^{pi_theta}(s_t,a_t) ] +``` + +Stochastic gradient ascent on policy performance: + +``` +theta_{k+1} = theta_k + alpha * grad_theta J(pi_theta_k) +``` + +**The notation mismatch worth knowing about:** `J(pi_theta)` here denotes the expected +**finite-horizon undiscounted** return, but policy gradient implementations typically compute +**advantage estimates based on the infinite-horizon discounted return** — despite otherwise +using the finite-horizon undiscounted policy gradient formula. This is the ch7 blurring of +the two return formulations, appearing in production code. + +## Frameworks Introduced +- **The VPG loop** (the template TRPO and PPO both modify, so learn it once): + 1. Collect a set of trajectories `D_k` by running `pi_k = pi(theta_k)` in the environment. + 2. Compute rewards-to-go `Rhat_t`. + 3. Compute advantage estimates `Ahat_t` (any method of advantage estimation) based on the + current value function `V_{phi_k}`. + 4. Estimate the policy gradient: + `ghat_k = (1/|D_k|) * sum_{tau in D_k} sum_t grad_theta log pi_theta(a_t|s_t)|_{theta_k} * Ahat_t` + 5. Update the policy by standard gradient ascent `theta_{k+1} = theta_k + alpha_k * ghat_k`, + or another gradient ascent algorithm like Adam. + 6. Fit the value function by regression on mean-squared error: + `phi_{k+1} = argmin_phi (1/(|D_k| T)) * sum sum (V_phi(s_t) - Rhat_t)^2`, typically via + gradient descent. + +## Exploration vs. Exploitation +VPG trains a **stochastic policy in an on-policy way**, so it explores by sampling actions +from the latest version of that policy. How random that is depends on both initial conditions +and the training procedure. **Over training the policy typically becomes progressively less +random**, because the update rule encourages exploiting rewards already found — **which may +cause the policy to get trapped in local optima.** (This paragraph is identical for TRPO and +PPO; it is a property of the on-policy stochastic-policy family, not of VPG specifically.) + +## Code Examples +Loading and using a trained PyTorch model: + +```python +ac = torch.load('path/to/model.pt') +actions = ac.act(torch.as_tensor(obs, dtype=torch.float32)) +``` + +The TF1 saved graph exposes three keys: `x` (state input placeholder), `pi` (samples an +action conditioned on `x`), and `v` (value estimate for states in `x`). + +## Anti-patterns +- **Expecting monotonic improvement.** VPG has none; the step size is unconstrained and a + bad step can collapse performance. That collapse is what TRPO (ch15) exists to prevent. +- **Under-fitting the value function.** Exercise 2.1 (ch12) shows a policy gradient agent + with an untrained value function gets stuck early. The advantage estimate is only as good + as `V_phi`. +- **Reading the loss.** See ch9's worked example. + +## Reference Tables + +| Relevant paper | Why it is on the list | +|----------------|----------------------| +| Sutton et al. 2000, *Policy Gradient Methods for RL with Function Approximation* | A timeless classic of RL theory; contains references to the earlier work that led to modern policy gradients | +| Schulman 2016(a), *Optimizing Expectations* | Chapter 2 is a lucid introduction to policy gradient theory, including pseudocode | +| Duan et al. 2016, *Benchmarking Deep RL for Continuous Control* | A clear benchmark paper showing how VPG in the deep RL setting compares with other deep RL algorithms | +| Schulman et al. 2016(b), *High Dimensional Continuous Control Using GAE* | Spinning Up's VPG uses GAE for computing the policy gradient | + +## Key Takeaways +1. VPG is the advantage-weighted policy gradient plus mean-squared value regression — nothing else. +2. The six-step loop is the base template; TRPO changes step 5, PPO changes step 5 differently. +3. Discrete or continuous, MPI-parallelizable, both backends available. +4. Its weakness is step-size safety: nothing stops a single update from collapsing the policy. +5. Exploration decays as a side effect of exploitation, with local optima the known risk. + +## Connects To +- **Ch 9**: the derivation that produces this exact gradient, and GAE. +- **Ch 15**: TRPO, which constrains the step this chapter leaves unconstrained. +- **Ch 12**: Exercise 2.1, on what happens when `V_phi` is not fit. +- **Ch 13**: the benchmark caveat — VPG is educational-grade, not research-grade. diff --git a/engineering/spinning-up-deep-rl/skills/spinning-up-deep-rl/chapters/ch15-trpo.md b/engineering/spinning-up-deep-rl/skills/spinning-up-deep-rl/chapters/ch15-trpo.md new file mode 100644 index 00000000..efbbb83a --- /dev/null +++ b/engineering/spinning-up-deep-rl/skills/spinning-up-deep-rl/chapters/ch15-trpo.md @@ -0,0 +1,122 @@ +# Chapter 15: Trust Region Policy Optimization (TRPO) + +## Core Idea +Take the **largest step possible to improve performance** while satisfying a constraint on +how close the new and old policies are — measured in **KL-divergence between policies**, not +distance in parameter space, because seemingly small parameter differences can produce very +large performance differences. + +## Quick Facts +- **On-policy.** +- Works with **discrete or continuous** action spaces. +- **Supports MPI parallelization.** +- **Tensorflow only** in Spinning Up; there is no PyTorch TRPO. + +## Key Equations +The theoretical update — maximize surrogate advantage subject to a KL trust region: + +``` +theta_{k+1} = argmax_theta L(theta_k, theta) + s.t. Dbar_KL(theta || theta_k) <= delta +``` + +The **surrogate advantage** measures how `pi_theta` performs relative to the old policy +*using data from the old policy* (an importance-weighted advantage): + +``` +L(theta_k, theta) = E_{s,a ~ pi_theta_k}[ (pi_theta(a|s) / pi_theta_k(a|s)) * A^{pi_theta_k}(s,a) ] + +Dbar_KL(theta || theta_k) = E_{s ~ pi_theta_k}[ D_KL( pi_theta(.|s) || pi_theta_k(.|s) ) ] +``` + +Both the objective and the constraint are **zero at `theta = theta_k`**, and the gradient of +the constraint is **also zero there**. + +## Frameworks Introduced +- **Taylor-expand to make it solvable.** Expand objective and constraint to leading order + around `theta_k`: + ``` + L(theta_k, theta) ~= g^T (theta - theta_k) + Dbar_KL(theta||theta_k) ~= 0.5 * (theta - theta_k)^T H (theta - theta_k) + ``` + **By happy coincidence, `g` — the gradient of the surrogate advantage at `theta_k` — is + exactly equal to the policy gradient `grad_theta J(pi_theta)`.** Lagrangian duality then + solves the approximate problem analytically: + ``` + theta_{k+1} = theta_k + sqrt( 2*delta / (g^T H^{-1} g) ) * H^{-1} g + ``` + **Stopping here would be exactly the Natural Policy Gradient.** + +- **Backtracking line search — the fix that makes it TRPO.** Because the Taylor expansion + introduces approximation error, that update may violate the KL constraint or fail to improve + the surrogate advantage. So: + ``` + theta_{k+1} = theta_k + alpha^j * sqrt( 2*delta / (g^T H^{-1} g) ) * H^{-1} g + ``` + where `alpha` in (0,1) is the **backtracking coefficient** and `j` is the **smallest + nonnegative integer** such that the new policy satisfies the KL constraint and produces a + positive surrogate advantage. + - When to use this pattern generally: whenever you solve an approximated problem and can + cheaply check the exact condition, shrink the step until the exact condition holds. + +- **Conjugate gradient instead of a matrix inverse.** Computing and storing `H^{-1}` is + painfully expensive for networks with thousands or millions of parameters. TRPO solves + `Hx = g` for `x = H^{-1} g` with conjugate gradient, which needs only a function computing + the matrix-vector product `Hx`, never `H` itself: + ``` + Hx = grad_theta( (grad_theta Dbar_KL(theta || theta_k))^T x ) + ``` + +## Reference Tables + +| Hyperparameter | Role | +|----------------|------| +| `delta` | KL-divergence limit — the size of the trust region | +| `alpha` | Backtracking coefficient, in (0,1) | +| `K` | Maximum number of backtracking steps; `j` ranges over {0, 1, ..., K} | + +The full loop is VPG's six steps with steps 5 replaced by: (a) conjugate gradient to compute +`xhat_k ~= Hhat_k^{-1} ghat_k`, where `Hhat_k` is the Hessian of the sample average +KL-divergence, then (b) the backtracking line search using +`theta_{k+1} = theta_k + alpha^j * sqrt(2 delta / (xhat_k^T Hhat_k xhat_k)) * xhat_k`. + +## Mental Models +- **Trust region in policy space, not parameter space.** This is the whole insight. Ordinary + policy gradient methods keep policies close in *parameter* space, but small parameter + differences can mean very large performance differences — **so a single bad step can + collapse the policy performance.** That is what makes large step sizes dangerous with + vanilla policy gradients, and it is why VPG's sample efficiency suffers: it has to take + small steps. +- **TRPO buys back sample efficiency by making large steps safe**, and tends to improve + performance **quickly and monotonically**. +- Read the algorithm as **three nested approximations** — surrogate objective, Taylor + expansion, conjugate gradient — each with a guard, the last being the line search. + +## Exploration vs. Exploitation +Identical to VPG: stochastic policy, on-policy sampling, randomness decaying over training as +the update rule pushes toward exploiting known rewards, with local optima as the risk. + +## Anti-patterns +- **Skipping the line search** and shipping the analytic solution. That is Natural Policy + Gradient, and the approximation error it inherits may violate the constraint or fail to + improve anything. +- **Forming `H` explicitly.** The matrix-vector product formulation exists because the matrix + does not fit. +- **Reaching for TRPO in PyTorch inside Spinning Up.** It does not exist; use PPO. + +## Key Takeaways +1. Constrain the *policy* change in KL, not the *parameter* change — the core contribution. +2. Surrogate advantage is importance-weighted old-policy data; both it and the KL constraint + vanish at `theta_k`. +3. `g` (surrogate advantage gradient) equals the policy gradient — that is why VPG's machinery + is reusable here. +4. Analytic solution = Natural Policy Gradient; TRPO adds a backtracking line search on top. +5. Conjugate gradient replaces the intractable `H^{-1}`. +6. Three relevant papers: Schulman 2015 (original), Schulman 2016 (GAE, used here), Kakade and + Langford 2002 (the theory motivating and connecting to TRPO's foundations). + +## Connects To +- **Ch 14**: the VPG loop TRPO modifies, and the collapse risk it removes. +- **Ch 16**: PPO, which pursues the same goal with first-order methods. +- **Ch 12**: Exercise 2.1 uses TRPO to demonstrate the value-fitting failure mode. +- **Ch 10**: "monotonic improvement theory" — the optional math background this rests on. diff --git a/engineering/spinning-up-deep-rl/skills/spinning-up-deep-rl/chapters/ch16-ppo.md b/engineering/spinning-up-deep-rl/skills/spinning-up-deep-rl/chapters/ch16-ppo.md new file mode 100644 index 00000000..6034a856 --- /dev/null +++ b/engineering/spinning-up-deep-rl/skills/spinning-up-deep-rl/chapters/ch16-ppo.md @@ -0,0 +1,123 @@ +# Chapter 16: Proximal Policy Optimization (PPO) + +## Core Idea +PPO answers the same question as TRPO — how do you take the biggest possible improvement +step without stepping so far that you cause performance collapse — but with **first-order +methods and a few tricks** instead of a complex second-order method. It is significantly +simpler to implement and empirically seems to perform at least as well. + +## Quick Facts +- **On-policy.** +- Works with **discrete or continuous** action spaces. +- **Supports MPI parallelization.** +- Two variants exist; Spinning Up implements and documents **PPO-Clip**, the primary variant + used at OpenAI. + +## Frameworks Introduced +- **The two PPO variants:** + - **PPO-Penalty** approximately solves a KL-constrained update like TRPO, but **penalizes + the KL-divergence in the objective** instead of making it a hard constraint, and + **automatically adjusts the penalty coefficient** over training so it stays appropriately + scaled. + - **PPO-Clip** has **no KL term in the objective and no constraint at all.** It relies on + specialized **clipping in the objective function** to remove the incentive for the new + policy to get far from the old. + +- **The PPO-Clip objective.** Update by + `theta_{k+1} = argmax_theta E_{s,a ~ pi_theta_k}[ L(s,a,theta_k,theta) ]`, typically with + **multiple steps of (usually minibatch) SGD**. The published form: + ``` + L = min( ratio * A, clip(ratio, 1-eps, 1+eps) * A ) + where ratio = pi_theta(a|s) / pi_theta_k(a|s) + ``` + and `eps` is a small hyperparameter roughly saying how far the new policy may go. + +- **The simplified equivalent form** (this is the version Spinning Up actually implements, + and it is much easier to reason about): + ``` + L = min( ratio * A, g(eps, A) ) + where g(eps, A) = (1 + eps) * A if A >= 0 + (1 - eps) * A if A < 0 + ``` + +## Worked Example +**Why clipping removes the incentive to move far — the two cases.** + +**Advantage positive.** The term reduces to `min(ratio, 1+eps) * A`. Because `A > 0`, the +objective increases as the action becomes more likely, i.e. as `pi_theta(a|s)` increases. +But the `min` caps how much. Once `pi_theta(a|s) > (1+eps) * pi_theta_k(a|s)`, the min kicks +in and the term hits a ceiling of `(1+eps) * A`. **The new policy does not benefit by going +far away from the old policy.** + +**Advantage negative.** The term reduces to `max(ratio, 1-eps) * A`. Because `A < 0`, the +objective increases as the action becomes *less* likely, i.e. as `pi_theta(a|s)` decreases. +The `max` caps how much: once `pi_theta(a|s) < (1-eps) * pi_theta_k(a|s)`, it hits a ceiling +of `(1-eps) * A`. **Again: the new policy does not benefit by going far away from the old.** + +So **clipping serves as a regularizer by removing incentives for the policy to change +dramatically**, and `eps` corresponds to how far the new policy can go while still profiting +from the objective. + +**The honest caveat, and Spinning Up's answer.** Clipping goes a long way toward reasonable +updates, but **it is still possible to end up with a new policy too far from the old**, and +different PPO implementations use a variety of tricks to stave this off. Spinning Up uses a +particularly simple one: **early stopping — if the mean KL-divergence of the new policy from +the old grows beyond a threshold, stop taking gradient steps.** Note that this reintroduces a +KL measurement into an algorithm defined by not having one; the clip is the incentive +mechanism, the KL check is the safety net. + +## Reference Tables + +The PPO-Clip loop, as a diff against VPG's six steps (ch14): + +| Step | VPG | PPO-Clip | +|------|-----|----------| +| 1-3 | Collect trajectories, rewards-to-go, advantage estimates | identical | +| 4-5 | Estimate `ghat_k`, single ascent step | **Maximize the PPO-Clip objective, typically via multiple steps of stochastic gradient ascent with Adam** | +| 6 | Fit `V_phi` by MSE regression on `Rhat_t` | identical | + +The key structural difference from VPG and TRPO: **multiple gradient steps per batch of data**, +which is where the sample-efficiency gain comes from and why the policy needs restraining at all. + +| Relevant paper | Why it is on the list | +|----------------|----------------------| +| Schulman et al. 2017, *Proximal Policy Optimization Algorithms* | The original PPO paper | +| Schulman et al. 2016, *High Dimensional Continuous Control Using GAE* | Spinning Up's PPO uses GAE for the policy gradient | +| Heess et al. 2017, *Emergence of Locomotion Behaviours in Rich Environments* | Large-scale empirical analysis of behaviors learned by PPO agents in complex environments — though it uses PPO-Penalty, not PPO-Clip | + +## Exploration vs. Exploitation +Identical to VPG and TRPO: stochastic policy, on-policy sampling, decaying randomness, local +optima as the failure mode. + +## Mental Models +- **PPO trades a hard guarantee for implementability.** TRPO enforces the trust region; + PPO removes the *incentive* to leave it. That is a weaker statement, and PPO's practical + dominance says the weaker statement is usually enough. +- **The clip is one-sided per case.** It only ever caps the *beneficial* direction. It never + penalizes moving the wrong way — it just stops paying you for moving further the right way. +- Careful with terminology: **ch9 notes PPO is often called a policy gradient algorithm, + though this is slightly inaccurate** — it maximizes a surrogate objective rather than + following `grad J` directly. + +## Anti-patterns +- **Implementing PPO-Clip and calling the result PPO without saying which variant.** Two + public implementations named "PPO" (ModularRL, rllab) implement PPO-**Penalty**; comparing + against them without noticing is a real reproduction hazard the docs explicitly flag. +- **Relying on the clip alone at large step counts.** More SGD steps per batch means more + opportunity to drift; that is precisely why the early-stopping KL check exists. +- **Tuning `eps` as if it were a learning rate.** It bounds policy change per update; the + number of SGD steps and the learning rate interact with it. + +## Key Takeaways +1. Same goal as TRPO, first-order machinery: simpler to implement, at least as good empirically. +2. PPO-Clip (no KL term, no constraint) is the OpenAI default and Spinning Up's implementation. +3. The simplified `g(eps, A)` form makes the mechanism obvious: a ceiling in the profitable + direction, in both the positive- and negative-advantage cases. +4. Clipping is an incentive change, not a guarantee — Spinning Up backs it with KL early stopping. +5. Multiple minibatch SGD steps per batch is the structural difference from VPG and TRPO. + +## Connects To +- **Ch 15**: TRPO's hard constraint, the thing PPO replaces. +- **Ch 9**: the surrogate-objective framing and why "policy gradient algorithm" is imprecise here. +- **Ch 12**: Exercise 1.2 asks you to implement the MLP diagonal Gaussian policy for PPO. +- **Ch 13**: use OpenAI Baselines' PPO, not this one, for research comparisons. diff --git a/engineering/spinning-up-deep-rl/skills/spinning-up-deep-rl/chapters/ch17-ddpg.md b/engineering/spinning-up-deep-rl/skills/spinning-up-deep-rl/chapters/ch17-ddpg.md new file mode 100644 index 00000000..75e296a3 --- /dev/null +++ b/engineering/spinning-up-deep-rl/skills/spinning-up-deep-rl/chapters/ch17-ddpg.md @@ -0,0 +1,122 @@ +# Chapter 17: Deep Deterministic Policy Gradient (DDPG) + +## Core Idea +DDPG concurrently learns a Q-function and a policy: it uses off-policy data and the Bellman +equation to learn the Q-function, and uses the Q-function to learn the policy. It exists +because `max_a Q*(s,a)` — trivial over discrete actions — is intractable over continuous ones, +and DDPG replaces that max with a learned, differentiable approximation. + +## Quick Facts +- **Off-policy.** +- **Continuous action spaces only.** +- Can be thought of as **deep Q-learning for continuous action spaces.** +- The Spinning Up implementation **does not support parallelization.** + +## Frameworks Introduced +- **The continuous-argmax problem, and the substitution that solves it.** With finitely many + discrete actions, the max poses no problem — compute Q for each and compare, which also + immediately hands you the maximizing action. With a continuous action space you cannot + exhaustively evaluate, and a general optimization subroutine would have to run **every time + the agent wants to take an action**, which is unacceptable. But because the action space is + continuous, `Q*(s,a)` is presumed **differentiable with respect to the action** — so you can + learn a policy `mu(s)` by gradient ascent and approximate + `max_a Q(s,a) ~= Q(s, mu(s))`. + - When to use this pattern: any time an inner argmax over a continuous variable sits inside + an outer loop. Amortize it into a learned function. + +- **Mean-Squared Bellman Error (MSBE)** — the loss that all deep Q-learning minimizes: + ``` + L(phi, D) = E_{(s,a,r,s',d) ~ D}[ ( Q_phi(s,a) - ( r + gamma*(1-d)*max_a' Q_phi(s',a') ) )^2 ] + ``` + `d` indicates whether `s'` is terminal, using the Python convention of `True` as 1: when + `d == True` the Q-function should show the agent gets no additional reward after this state. + +- **Trick One: Replay Buffers.** All standard algorithms training a deep network to approximate + `Q*(s,a)` use an experience replay buffer — the set `D` of previous experiences. + - **The buffer size is a real tuning decision, in both directions:** it should be large + enough to contain a wide range of experiences, but keeping everything may not be good + either. **Use only the very-most recent data and you overfit to it and things break; use + too much experience and you may slow down learning.** + - **Why old data is admissible at all** — this is the cleanest statement of what off-policy + means: **the Bellman equation does not care which transition tuples are used, how the + actions were selected, or what happens after a given transition**, because the optimal + Q-function must satisfy it for *all* possible transitions. So any transition you have ever + experienced is fair game for MSBE minimization. + +- **Trick Two: Target Networks.** The target `r + gamma*(1-d)*max_a' Q_phi(s',a')` depends on + the same parameters `phi` being trained, which makes MSBE minimization unstable. The fix is + a second network that lags the first, with parameters `phi_targ`. + - **DQN-style**: copy the main network into the target every fixed number of steps. + - **DDPG-style**: **polyak averaging**, once per main network update: + `phi_targ <- rho * phi_targ + (1-rho) * phi`, with `rho` in (0,1), usually close to 1. + (`rho` is called `polyak` in the code.) + +- **DDPG detail: the max in the target.** Since the max over continuous actions is the original + problem, DDPG uses a **target policy network** `mu_theta_targ` to compute an action that + approximately maximizes `Q_phi_targ`, found the same way as the target Q-function — by + polyak averaging the policy parameters over training. Final Q-loss: + ``` + L(phi,D) = E[ ( Q_phi(s,a) - ( r + gamma*(1-d)*Q_phi_targ(s', mu_theta_targ(s')) ) )^2 ] + ``` + +- **Policy learning is one line**: maximize `E_{s~D}[ Q_phi(s, mu_theta(s)) ]` by gradient + ascent **with respect to policy parameters only** — the Q-function parameters are treated + as constants. + +## Exploration vs. Exploitation +DDPG trains a **deterministic policy off-policy**. Because the policy is deterministic, +on-policy exploration would probably not try a wide enough variety of actions early on to +find useful learning signal. So **noise is added to actions at training time**. +- The original DDPG paper recommended **time-correlated OU noise**, but **more recent results + suggest uncorrelated, mean-zero Gaussian noise works perfectly well** — and since it is + simpler, it is preferred. +- You may reduce the noise scale over training to get higher-quality data. **Spinning Up does + not do this and keeps the noise scale fixed throughout.** +- **At test time, no noise is added** — that is how you see how well the policy exploits what + it has learned. +- **`start_steps` trick**: for a fixed number of steps at the beginning, the agent takes + actions sampled from a **uniform random distribution over valid actions**, then reverts to + normal DDPG exploration. + +## Reference Tables + +The DDPG loop: + +| Step | Action | +|------|--------| +| Init | Set target parameters equal to main: `theta_targ <- theta`, `phi_targ <- phi` | +| Act | Observe `s`, select `a = clip(mu_theta(s) + epsilon, a_Low, a_High)`, `epsilon ~ N` | +| Store | Execute `a`, observe `s'`, `r`, `d`; store `(s,a,r,s',d)` in `D`; reset if terminal | +| Update | Sample a batch `B` from `D`; compute targets `y = r + gamma*(1-d)*Q_phi_targ(s', mu_theta_targ(s'))` | +| | One gradient descent step on `(Q_phi(s,a) - y)^2` averaged over `B` | +| | One gradient ascent step on `Q_phi(s, mu_theta(s))` averaged over `B` | +| | Polyak-update both target networks | + +| Relevant paper | Why it is on the list | +|----------------|----------------------| +| Silver et al. 2014, *Deterministic Policy Gradient Algorithms* | Establishes the theory underlying deterministic policy gradients (DPG) | +| Lillicrap et al. 2016, *Continuous Control With Deep RL* | Adapts the theoretically-grounded DPG algorithm to the deep RL setting, giving DDPG | + +## Anti-patterns +- **DDPG on a discrete action space.** The entire design is the continuous-action workaround. +- **Trusting the original paper's full recipe.** Ch10 uses DDPG as its example: the paper + suggests a complex architecture, an initialization scheme and batch normalization, none of + which are strictly necessary, and some of the best DDPG results use simpler networks. +- **Assuming DDPG is stable.** It is **frequently brittle with respect to hyperparameters and + other kinds of tuning** — that is the premise of ch18. +- **Shape bugs in the Q-function.** Exercise 2.2 (ch12) plants exactly one in DDPG's critic + and it silently ruins learning through broadcasting. + +## Key Takeaways +1. `Q(s, mu(s))` replaces `max_a Q(s,a)` — the whole reason DDPG exists. +2. Replay buffers are licensed by the Bellman equation's indifference to how data was collected. +3. Target networks fix the moving-target instability; DDPG uses polyak averaging, DQN uses copies. +4. Two target networks in DDPG: target Q *and* target policy. +5. Explore with additive Gaussian action noise plus uniform-random `start_steps`; no noise at test. +6. Replay buffer size is a genuine trade-off, wrong in both directions. + +## Connects To +- **Ch 7**: `a*(s) = argmax_a Q*(s,a)` and the Bellman optimality equation this is built on. +- **Ch 8**: DDPG as the named interpolation between policy optimization and Q-learning. +- **Ch 18**: TD3, the three tricks that fix DDPG's brittleness. +- **Ch 12**: Exercise 2.2, the silent bug planted in this algorithm. diff --git a/engineering/spinning-up-deep-rl/skills/spinning-up-deep-rl/chapters/ch18-td3.md b/engineering/spinning-up-deep-rl/skills/spinning-up-deep-rl/chapters/ch18-td3.md new file mode 100644 index 00000000..dae89274 --- /dev/null +++ b/engineering/spinning-up-deep-rl/skills/spinning-up-deep-rl/chapters/ch18-td3.md @@ -0,0 +1,107 @@ +# Chapter 18: Twin Delayed DDPG (TD3) + +## Core Idea +DDPG's common failure mode is that **the learned Q-function begins to dramatically +overestimate Q-values, which then breaks the policy, because the policy exploits the errors +in the Q-function.** TD3 fixes this with three tricks, and nothing else changes. + +## Quick Facts +- **Off-policy.** +- **Continuous action spaces only.** +- The Spinning Up implementation **does not support parallelization.** + +## Frameworks Introduced +- **Trick One: Clipped Double-Q Learning.** Learn **two** Q-functions instead of one (hence + "twin") and **use the smaller of the two Q-values to form the targets** in the Bellman error + loss functions. Both are then regressed toward that single shared target: + ``` + y(r,s',d) = r + gamma*(1-d) * min_{i=1,2} Q_{phi_i,targ}(s', a'(s')) + L(phi_1, D) = E[ (Q_phi_1(s,a) - y)^2 ] + L(phi_2, D) = E[ (Q_phi_2(s,a) - y)^2 ] + ``` + **Using the smaller Q-value for the target, and regressing toward that, helps fend off + overestimation** — a pessimistic estimate cannot be exploited the way an optimistic one can. + +- **Trick Two: "Delayed" Policy Updates.** Update the policy (and the target networks) **less + frequently than the Q-function**. **The paper recommends one policy update for every two + Q-function updates.** This **damps the volatility that normally arises in DDPG because of + how a policy update changes the target.** Controlled by `policy_delay`. + +- **Trick Three: Target Policy Smoothing.** Add **clipped** noise to each dimension of the + target action, then clip the result back into the valid action range: + ``` + a'(s') = clip( mu_theta_targ(s') + clip(epsilon, -c, c), a_Low, a_High ), epsilon ~ N(0, sigma) + ``` + **This is a regularizer.** It addresses a specific DDPG failure: **if the Q-function + approximator develops an incorrect sharp peak for some actions, the policy will quickly + exploit that peak and then have brittle or incorrect behavior.** Smoothing Q over similar + actions averts it. + +- **Policy learning is unchanged from DDPG except in which critic it uses:** maximize + `E_{s~D}[ Q_phi_1(s, mu_theta(s)) ]` — **just the first Q-function.** (SAC differs here; see + ch19.) + +## Reference Tables + +| Trick | Fixes | Mechanism | +|-------|-------|-----------| +| Clipped double-Q | Q-value overestimation | Two critics, `min` of the two forms the shared target | +| Delayed policy updates | Volatility from policy updates moving the target | One policy + target update per `policy_delay` critic updates | +| Target policy smoothing | Policy exploiting sharp incorrect peaks in Q | Clipped Gaussian noise on the target action, then clip to action bounds | + +| Hyperparameter | Role | +|----------------|------| +| `sigma` | Std dev of the target-smoothing noise | +| `c` | Clip bound on that noise | +| `policy_delay` | Q-updates per policy update; paper recommends 2 | +| `rho` (`polyak`) | Target network averaging coefficient, inherited from DDPG | +| `start_steps` | Initial uniform-random action steps, inherited from DDPG | + +The loop is DDPG's, with two changes inside the update block: compute the smoothed target +action `a'(s')` first, update **both** Q-functions toward the clipped-double-Q target, and +then — **only when `j mod policy_delay == 0`** — take the policy ascent step and polyak-update +all three target networks. + +## Exploration vs. Exploitation +Identical to DDPG: deterministic policy explored with **uncorrelated mean-zero Gaussian action +noise** at training time, optionally decayed (Spinning Up keeps it fixed), **no noise at test +time**, and a `start_steps` phase of uniform random actions at the beginning. + +Note the distinction that is easy to blur: **exploration noise is added to the acting policy; +target policy smoothing noise is added to the target action inside the update.** Two different +noises, two different jobs. + +## Mental Models +- **All three tricks are forms of pessimism or patience.** Take the smaller estimate, wait + longer before acting on the critic, and refuse to believe a Q-value that does not hold up + under a small action perturbation. Overestimation is the disease; conservatism is the cure. +- **Think of the policy as an adversary against your critic's errors.** Anything the critic + gets wrong in the optimistic direction is exactly what the policy will find. This is the + same failure shape as model bias in ch8's model-based methods. +- **"Together, these three tricks result in substantially improved performance over baseline + DDPG"** — they are presented as a package, and the exercise (ch12, 1.3) has you implement + the losses that contain all three. + +## Anti-patterns +- **Implementing only clipped double-Q and calling it TD3.** The three tricks address three + distinct failure modes and the paper's result is for the package. +- **Using `min` over the two critics for the *policy* loss.** TD3 uses `Q_phi_1` alone for the + policy; using the min there is a SAC choice, not a TD3 one. +- **Updating the target networks on every critic step.** The target updates are inside the + `policy_delay` branch, along with the policy update. + +## Key Takeaways +1. TD3 = DDPG + clipped double-Q + delayed policy updates + target policy smoothing. +2. The disease being treated is Q-value overestimation, which the policy actively exploits. +3. `min` of two critics forms the target; the policy maximizes `Q_phi_1` only. +4. Policy and target-network updates both live behind `policy_delay` (recommended 2). +5. Target smoothing noise is clipped twice: clip the noise to `[-c, c]`, then clip the + resulting action to the valid range. +6. Relevant paper: Fujimoto et al, 2018, *Addressing Function Approximation Error in + Actor-Critic Methods.* + +## Connects To +- **Ch 17**: DDPG, whose brittleness is TD3's entire premise. +- **Ch 19**: SAC, which borrows clipped double-Q and gets smoothing for free from stochasticity. +- **Ch 12**: Exercise 1.3 asks you to write exactly these losses. +- **Ch 13**: TD3 is one of the three research-grade Spinning Up implementations. diff --git a/engineering/spinning-up-deep-rl/skills/spinning-up-deep-rl/chapters/ch19-sac.md b/engineering/spinning-up-deep-rl/skills/spinning-up-deep-rl/chapters/ch19-sac.md new file mode 100644 index 00000000..d3a36041 --- /dev/null +++ b/engineering/spinning-up-deep-rl/skills/spinning-up-deep-rl/chapters/ch19-sac.md @@ -0,0 +1,133 @@ +# Chapter 19: Soft Actor-Critic (SAC) + +## Core Idea +SAC optimizes a **stochastic** policy in an **off-policy** way, forming a bridge between +stochastic policy optimization and DDPG-style approaches. Its central feature is **entropy +regularization**: the policy is trained to maximize a trade-off between expected return and +the randomness of the policy. + +## Quick Facts +- **Off-policy.** +- The version implemented here is **continuous action spaces only** — an alternate version, + with a slightly changed policy update rule, can handle discrete action spaces. +- **Does not support parallelization.** +- SAC is **not a direct successor to TD3** — they were published roughly concurrently — but it + incorporates the clipped double-Q trick, and its inherent policy stochasticity gives it + something like target policy smoothing for free. + +## Frameworks Introduced +- **Entropy-regularized RL.** Entropy `H(P) = E_{x~P}[-log P(x)]` says how random a random + variable is: a coin that almost always comes up heads has low entropy, a fair coin has high + entropy. The agent gets a bonus reward at each timestep proportional to policy entropy, + changing the RL problem to: + ``` + pi* = argmax_pi E_{tau~pi}[ sum_t gamma^t ( R(s_t,a_t,s_{t+1}) + alpha * H(pi(.|s_t)) ) ] + ``` + `alpha > 0` is the trade-off coefficient. **This has a close connection to the + exploration-exploitation trade-off: increasing entropy results in more exploration, which + can accelerate learning later on, and can prevent the policy from prematurely converging to + a bad local optimum.** + +- **The modified value functions.** `V^pi` includes the entropy bonus from **every** timestep; + `Q^pi` includes it from every timestep **except the first**. They connect by + ``` + V^pi(s) = E_{a~pi}[ Q^pi(s,a) ] + alpha * H(pi(.|s)) + = E_{a~pi}[ Q^pi(s,a) - alpha * log pi(a|s) ] + ``` + **This setup is a little bit arbitrary** — you could instead have `Q^pi` include the first + timestep's bonus — and **the choice of definition varies slightly across papers on the + subject.** Check before comparing equations across sources. + +- **What SAC keeps from TD3, and what it changes.** SAC learns a policy `pi_theta` and two + Q-functions `Q_phi_1`, `Q_phi_2`. + - **Same as TD3**: both Q-functions learned by MSBE minimization regressing to a **single + shared target**; the target computed using **target Q-networks obtained by polyak + averaging**; the **clipped double-Q trick**. + - **Different from TD3**: (1) the target includes an **entropy regularization term**; + (2) the **next-state actions in the target come from the current policy, not a target + policy**; (3) **no explicit target policy smoothing** — TD3 trains a deterministic policy + and so needs added noise to smooth, whereas **SAC's policy stochasticity is sufficient to + get a similar effect**. + +- **The Q-loss.** Rewrite the entropy-regularized Bellman equation using + `H = -log pi`, approximate the expectation with samples, and take the min over the two + critics: + ``` + y(r,s',d) = r + gamma*(1-d) * ( min_{j=1,2} Q_{phi_j,targ}(s', a'~) - alpha * log pi_theta(a'~|s') ) + where a'~ ~ pi_theta(.|s') + ``` + **Notation matters here:** `r` and `s'` come from the **replay buffer**, but `a'~` must be + **sampled fresh from the current policy** — hence the tilde. Mixing these up is a silent bug. + +- **Policy learning via the reparameterization trick.** The policy should maximize `V^pi(s)`, + i.e. `E_{a~pi}[ Q^pi(s,a) - alpha log pi(a|s) ]`. The expectation's distribution depends on + the policy parameters, which is the pain point; the reparameterization trick rewrites it as + an expectation over **noise**, which has no parameter dependence. Using a **squashed Gaussian + policy**: + ``` + a~_theta(s, xi) = tanh( mu_theta(s) + sigma_theta(s) * xi ), xi ~ N(0, I) + ``` + giving the policy objective + ``` + max_theta E_{s~D, xi~N}[ min_{j=1,2} Q_phi_j(s, a~_theta(s,xi)) - alpha * log pi_theta(a~_theta(s,xi)|s) ] + ``` + **almost the same as DDPG and TD3 policy optimization, except for the min-double-Q trick, + the stochasticity, and the entropy term.** Note that unlike TD3 (which uses `Q_phi_1` only), + **SAC uses the min of the two approximators in the policy loss too.** + +## Reference Tables + +**Two ways SAC's policy differs from the VPG/TRPO/PPO policies** — both are load-bearing: + +| Difference | SAC | VPG / TRPO / PPO | +|-----------|-----|------------------| +| **Squashing function** | `tanh` ensures actions are bounded to a finite range. Before the tanh the policy is a factored Gaussian; after it, it is not. Log-probabilities are still computable in closed form (see the paper appendix) | No squashing | +| **Std dev parameterization** | log stds are **outputs of the neural network**, so they depend on state in a complex way | log stds are **state-independent parameter vectors** | + +**SAC with state-independent log std devs, in the authors' experience, did not work.** + +| Variant | Status | +|---------|--------| +| Fixed entropy coefficient `alpha` | What Spinning Up implements, for simplicity | +| Entropy-constrained (varies `alpha` over training) | **Generally preferred by practitioners** | +| Older SAC that also learns a value function `V_psi` | Superseded; the docs cover the modern version that omits it | + +## Exploration vs. Exploitation +SAC trains a stochastic policy with entropy regularization and explores **on-policy**. +**`alpha` explicitly controls the explore-exploit trade-off** — higher means more exploration, +lower means more exploitation. **The right coefficient may vary from environment to environment +and could require careful tuning.** At test time, **remove the stochasticity and use the mean +action instead of a sample**; this tends to improve performance over the stochastic policy. +(This is why `test_policy` has a SAC-only `--deterministic` flag — see ch5.) `start_steps` +uniform-random exploration at the beginning applies here too. + +## Anti-patterns +- **Evaluating SAC with the stochastic policy.** The correct evaluation policy is the + deterministic mean; `test_policy` defaults to the stochastic one and needs `-d`. +- **Using state-independent log stds in SAC** because that is what PPO does. It did not work. +- **Sampling the target's next action from the replay buffer.** It must come fresh from the + current policy. +- **Assuming SAC's `Q^pi` definition matches another paper's.** The placement of the + first-timestep entropy bonus is a convention that varies. +- **Treating `alpha` as a set-and-forget constant across environments.** It is the + explore-exploit dial and is environment-specific; the entropy-constrained variant exists + because tuning it by hand is hard. + +## Key Takeaways +1. Entropy regularization is the defining feature: maximize return plus `alpha` times policy + entropy, which directly controls exploration. +2. SAC keeps TD3's clipped double-Q and polyak targets, adds an entropy term to the target, + uses the current policy (not a target policy) for next-state actions, and needs no explicit + smoothing. +3. The squashed Gaussian plus reparameterization trick makes the stochastic policy + differentiable end to end. +4. SAC uses `min` over both critics in the policy loss; TD3 uses only `Q_phi_1`. +5. State-dependent log stds are required — state-independent ones did not work. +6. Evaluate with the mean action. Spinning Up ships the fixed-`alpha` variant; practitioners + generally prefer the entropy-constrained one. + +## Connects To +- **Ch 18**: TD3, whose clipped double-Q trick SAC borrows and whose smoothing it obviates. +- **Ch 7**: the diagonal Gaussian policy and the two ways of parameterizing log stds. +- **Ch 5**: the SAC-only `--deterministic` flag on `test_policy`. +- **Ch 10**: the reparameterization trick is on the required deep-learning background list. diff --git a/engineering/spinning-up-deep-rl/skills/spinning-up-deep-rl/chapters/ch20-logger-and-utilities.md b/engineering/spinning-up-deep-rl/skills/spinning-up-deep-rl/chapters/ch20-logger-and-utilities.md new file mode 100644 index 00000000..fef7b3f7 --- /dev/null +++ b/engineering/spinning-up-deep-rl/skills/spinning-up-deep-rl/chapters/ch20-logger-and-utilities.md @@ -0,0 +1,99 @@ +# Chapter 20: Logger, MPI Tools and Run Utils + +## Core Idea +The three utility modules are the only code shared between algorithms — logging (which is +also model saving and config recording), MPI data-parallelism, and ExperimentGrid — and +each has a small fixed usage pattern worth memorizing. + +## Frameworks Introduced +- **`Logger` and `EpochLogger`.** `Logger` carries the basic functionality: saving + diagnostics, hyperparameter configurations, the state of a training run, and the trained + model. `EpochLogger` adds a thin layer that makes it easy to track the **average, standard + deviation, min and max** of a diagnostic over each epoch **and across MPI workers**. + **All Spinning Up algorithm implementations use an EpochLogger.** + - The three-call pattern: **`store`** accumulates values into internal state, + **`log_tabular`** computes the statistics over everything stored, **`dump_tabular`** + writes to file and stdout. **The internal state is wiped clean after `log_tabular`, + to prevent leakage into the next epoch's statistics.** + +- **The MPI + PyTorch pattern** — three steps, in this order: + 1. At the beginning of the training script, call **`setup_pytorch_for_mpi()`**. This is the + fix for a real problem: each separate process's PyTorch instance tries to grab too many + threads and they clobber each other. + 2. After constructing a PyTorch module, call **`sync_params(module)`**. + 3. During gradient descent, call **`mpi_avg_grads`** after the backward pass and before the + optimizer step. + - The two main ingredients are therefore **syncing parameters** and **averaging gradients + before they are used by the adaptive optimizer** — the order matters, because averaging + after the optimizer step would give each worker a different adaptive state. + +- **ExperimentGrid** — a tool for hyperparameter ablations, based on but simpler than rllab's + VariantGenerator. `eg.add(param_name, values, shorthand, in_name)` then + `eg.run(thunk, **run_kwargs)`; `ExperimentGrid.run` uses `call_experiment` to launch the + thunk. See ch4 for the full usage. + +## Code Examples +The EpochLogger statistic pattern: + +```python +from spinup.utils.logx import EpochLogger +epoch_logger = EpochLogger() +for i in range(10): + epoch_logger.store(Test=i) +epoch_logger.log_tabular('Test', with_min_and_max=True) +epoch_logger.dump_tabular() +``` + +``` +------------------------------------- +| AverageTest | 4.5 | +| StdTest | 2.87 | +| MaxTest | 9 | +| MinTest | 0 | +------------------------------------- +``` + +The MPI gradient-averaging step, in place: + +```python +optimizer.zero_grad() +loss = compute_loss(module) +loss.backward() +mpi_avg_grads(module) # averages gradient buffers across MPI processes +optimizer.step() +``` + +## Reference Tables + +| Module | Contents | +|--------|----------| +| `spinup.utils.logx` | `Logger`, `EpochLogger` | +| `spinup.utils.mpi_tools` | Core MPI utilities | +| `spinup.utils.mpi_pytorch` | `setup_pytorch_for_mpi`, `sync_params`, `mpi_avg_grads` | +| `spinup.utils.mpi_tf` | AdamOptimizer across MPI processes. **Explicitly "a bit hacky"** — for something more sophisticated and general-purpose, the docs point to horovod | +| `spinup.utils.run_utils` | `ExperimentGrid`, `call_experiment`, `setup_logger_kwargs` | +| `spinup.utils.plot` | The plotter (documented in ch6) | + +## Anti-patterns +- **Calling `mpi_avg_grads` after `optimizer.step()`.** The averaging must happen before the + adaptive optimizer consumes the gradients, or workers diverge in optimizer state. +- **Skipping `setup_pytorch_for_mpi()`.** The thread-clobbering problem it fixes is a real + performance failure, not a theoretical one. +- **Expecting `log_tabular` to be idempotent.** It wipes the accumulated state; calling it + twice for the same key in one epoch gives you statistics over nothing. +- **Reaching for `mpi_tf` as a general distributed-training solution.** The docs say to use + horovod instead if you need something serious. + +## Key Takeaways +1. `store` / `log_tabular` / `dump_tabular` is the whole logger interface, and the state + resets on `log_tabular`. +2. `EpochLogger` aggregates across MPI workers as well as across an epoch. +3. MPI PyTorch is three calls in a fixed order: setup, sync params, average grads before step. +4. The logger is also the model-saving and config-recording path (ch5). +5. `mpi_tf` is acknowledged as hacky; horovod is the recommended alternative. + +## Connects To +- **Ch 4**: ExperimentGrid usage and the CLI it backs. +- **Ch 5**: the outputs the logger writes — `progress.txt`, `config.json`, the save directories. +- **Ch 6**: the plotter, which reads what `dump_tabular` writes. +- **Ch 2**: OpenMPI, the system dependency all of this rests on. diff --git a/engineering/spinning-up-deep-rl/skills/spinning-up-deep-rl/cheatsheet.md b/engineering/spinning-up-deep-rl/skills/spinning-up-deep-rl/cheatsheet.md new file mode 100644 index 00000000..0970d31a --- /dev/null +++ b/engineering/spinning-up-deep-rl/skills/spinning-up-deep-rl/cheatsheet.md @@ -0,0 +1,87 @@ +# Cheatsheet + +Decision rules, thresholds and tells from Spinning Up. One page; keep it beside you. + +## Which algorithm? + +| If | Then | Because | +|----|------|---------| +| Discrete actions | PPO (or VPG to learn) | DDPG, TD3 and shipped SAC are continuous-only | +| Continuous, stability matters most | PPO | Directly optimizes performance; on-policy is stable | +| Continuous, sample efficiency matters most | SAC, else TD3 | Off-policy data reuse; both are research-grade here | +| You are learning the field | VPG first, then DQN, A2C, PPO, DDPG | Simplest first; complexity added gradually | +| You need a research baseline for TRPO/PPO | OpenAI Baselines, not Spinning Up | Spinning Up's on-policy trio omits observation normalization and normalized value targets | +| You need a research baseline for DDPG/TD3/SAC | Spinning Up is fine | Roughly at parity with best reported results | +| Partial observability or pixels | None of these as shipped | All six are non-recurrent MLP actor-critics | + +## Debugging a run that does not learn + +1. **Assume it is a bug, not the hyperparameters.** Bad hyperparameters degrade performance; + if yours resemble the paper's, they are probably not the cause. +2. **Check tensor shapes.** A `[N]` vs `[N,1]` mismatch is broadcast-compatible, raises + nothing, and silently corrupts the Bellman backup into an `[N,N]` matrix. +3. **Check what your loss is computed on** — wrong equation, wrong distribution, or data + piped to the wrong place are the three usual causes. +4. **Do not read the policy-gradient loss as a health signal.** Only average return means + anything; the loss can go to negative infinity while performance craters. +5. **Instrument more.** Mean/std/min/max of returns, episode lengths and value estimates; + objective losses; exploration parameters (policy entropy, epsilon). Watch videos too. +6. **Test in more than one environment** once results look promising — code can work in one + environment despite a breaking bug. + +## Thresholds and defaults + +| Quantity | Value | Source | +|----------|-------|--------| +| Debug-stage turnaround target | **under 5 minutes** locally | Ch 10 | +| Random seeds, minimum | **3** | Ch 10 | +| Random seeds, thorough | **10 or more** | Ch 10, Ch 13 | +| Debug environments | CartPole-v0, InvertedPendulum-v0, FrozenLake-v0, HalfCheetah-v2 at 100-250 steps | Ch 10 | +| From-scratch implementation size | ~250-300 lines; no-frills VPG ~80 | Ch 10 | +| On-policy benchmark network | (64, 32), tanh | Ch 13 | +| Off-policy benchmark network | (256, 256), relu | Ch 13 | +| On-policy batch | 4000 env steps per update | Ch 13 | +| Off-policy minibatch | 100 per gradient step | Ch 13 | +| TD3 `policy_delay` | 2 | Ch 18 | +| Polyak `rho` | in (0,1), usually close to 1 | Ch 17 | +| Benchmark length | 3M timesteps, 5 MuJoCo envs | Ch 13 | +| Plot smoothing used in the book | 11-epoch window | Ch 13 | + +## Choosing a research frame + +| Frame | Scope | Wraps up in | Main risk | +|-------|-------|-------------|-----------| +| Improve an existing approach | Narrow | A few months | Tweaks fail and you have no signal on what next | +| Unsolved benchmark | Broad | Months to a year-plus | May need a breakthrough; long time, no progress | +| New problem setting | Open-ended | Unbounded | You must design the benchmark too; cannot go looking for these | + +## Rigor checklist before reporting + +- [ ] Baseline tuned **as much as** your own method. Never handicap it. +- [ ] All else held equal (e.g. comparable parameter counts across architecture variants). +- [ ] At least 3 seeds, ideally 10 or more; plot mean and std dev. +- [ ] Final runs launched fresh and **precommitted** — not the best or most interesting runs. +- [ ] One ablation per design decision, so each claim is separately supported. +- [ ] Checked the idea has not already been published. + +## Tells and smells + +- **A learning curve with no seed band** is telling you less than it looks like. +- **`AverageEpRet` on an off-policy algorithm** is the noisy behavior policy, not performance. + Use the plotter's `Performance` alias. +- **Evaluating SAC without `--deterministic`** measures the wrong policy. +- **A model-based agent that scores well but behaves badly** is exploiting model bias — the + same shape as a policy exploiting Q-function errors. +- **"Fits the Bellman equation well"** carries no guarantee of good policy performance. +- **A single missing `squeeze`** in a critic is the archetype of silent failure. +- **A paper's full trick list** is usually more than strictly necessary — try simplifications. +- **A library abstraction** is good for reuse across algorithms and a cost when learning one. + +## Quick commands + +```bash +python -m spinup.run ppo --env Walker2d-v2 --exp_name walker --seed 0 10 20 +python -m spinup.run test_policy data/walker/walker_s0 # add -d for SAC +python -m spinup.run plot data/walker # prefix autocompletes +python -m spinup.run [algo] --help # every kwarg is a flag +``` diff --git a/engineering/spinning-up-deep-rl/skills/spinning-up-deep-rl/glossary.md b/engineering/spinning-up-deep-rl/skills/spinning-up-deep-rl/glossary.md new file mode 100644 index 00000000..7c2832fb --- /dev/null +++ b/engineering/spinning-up-deep-rl/skills/spinning-up-deep-rl/glossary.md @@ -0,0 +1,56 @@ +# Glossary + +**Action space** — the set of all valid actions in an environment; discrete or continuous (Ch 7) +**Action-value function** — see Q-function (Ch 7) +**Advantage function** — `A^pi(s,a) = Q^pi(s,a) - V^pi(s)`; how much better a specific action is than randomly selecting one per pi (Ch 7, Ch 9) +**Agent** — the learner and decider; often used interchangeably with "policy" (Ch 7) +**Backtracking line search** — shrink the step by `alpha^j` until the exact KL constraint holds and surrogate advantage is positive (Ch 15) +**Baseline** — any state-only function `b(s)` subtractable from the policy gradient weight without changing its expectation (Ch 9) +**Bellman backup** — the right-hand side of a Bellman equation: reward plus next value (Ch 7) +**Bellman equation** — self-consistency condition: the value of your starting point is the reward you expect from being there plus the value of where you land next (Ch 7) +**Categorical policy** — stochastic policy for discrete actions, built like a classifier: logits then softmax (Ch 7) +**Clipped double-Q learning** — learn two Q-functions and use the smaller value to form the target (Ch 18, Ch 19) +**Deadly triad** — function approximation + bootstrapping + off-policy data, jointly causing instability in value learning (Ch 8) +**Diagonal Gaussian policy** — stochastic policy for continuous actions; mean network plus a diagonal covariance represented as a vector of log stds (Ch 7) +**Discount factor (gamma)** — in (0,1); makes an infinite reward sum converge and encodes "cash now beats cash later" (Ch 7) +**EGLP lemma** — Expected Grad-Log-Prob: `E_{x~P_theta}[grad log P_theta(x)] = 0` (Ch 9) +**Entropy** — `H(P) = E_{x~P}[-log P(x)]`; how random a random variable is (Ch 19) +**Entropy regularization** — adding an entropy bonus to the reward at each timestep, scaled by `alpha` (Ch 19) +**Epoch** — one experience-collection phase plus one policy update (Ch 9) +**ExperimentGrid** — Spinning Up's hyperparameter ablation tool; backs the CLI (Ch 4, Ch 20) +**Fully / partially observed** — whether the agent sees the complete state or only part of it (Ch 7) +**GAE** — Generalized Advantage Estimation; used by Spinning Up's VPG, TRPO and PPO (Ch 9) +**Log-derivative trick** — `grad P = P * grad log P` (Ch 9) +**Markov property** — transitions depend only on the most recent state and action (Ch 7) +**MDP** — Markov Decision Process, the 5-tuple `` (Ch 7) +**Model** — a function predicting state transitions and rewards (Ch 8) +**Model-based / model-free** — whether the agent has or learns a model (Ch 8) +**MPC** — model-predictive control: plan over a window, execute the first action, discard the rest, re-plan (Ch 8) +**MSBE** — Mean-Squared Bellman Error, the loss deep Q-learning minimizes (Ch 17) +**Natural Policy Gradient** — TRPO's analytic solution before the line search is added (Ch 15) +**Observation** — a partial description of a state, possibly omitting information (Ch 7) +**Off-policy** — updates may use data collected at any point in training (Ch 3, Ch 8) +**On-policy** — updates use only data from the most recent policy (Ch 3, Ch 8) +**Optimal policy (pi\*)** — `argmax_pi J(pi)` (Ch 7) +**Parameterized policy** — a policy whose output depends on adjustable parameters, e.g. network weights (Ch 7) +**Performance** — the plotter's family-aware alias, resolving to `AverageEpRet` (on-policy) or `AverageTestEpRet` (off-policy) (Ch 6) +**Policy** — the rule for choosing actions; deterministic `mu(s)` or stochastic `pi(.|s)` (Ch 7) +**Policy delay** — TD3's ratio of Q-updates to policy updates; recommended 2 (Ch 18) +**Policy gradient** — `grad_theta J(pi_theta)` (Ch 9) +**Polyak averaging** — target update `phi_targ <- rho*phi_targ + (1-rho)*phi`; `polyak` in the code (Ch 17) +**Q-function** — `Q^pi(s,a)`, expected return from taking a in s then following pi (Ch 7) +**Replay buffer** — the stored set of past transitions that off-policy algorithms sample from (Ch 17) +**Reparameterization trick** — sample by a deterministic function of state, parameters and independent noise (Ch 19) +**Return** — cumulative reward over a trajectory; finite-horizon undiscounted or infinite-horizon discounted (Ch 7) +**Reward-to-go** — `Rhat_t = sum_{t'=t}^T R(s_t', a_t', s_{t'+1})` (Ch 9) +**Silent failure** — broken RL code that runs correctly and simply never learns (Ch 10, Ch 12) +**Squashed Gaussian policy** — a Gaussian passed through `tanh` to bound actions; SAC's policy (Ch 19) +**start_steps** — initial phase of uniform-random actions used by DDPG, TD3 and SAC (Ch 17) +**State** — a complete description of the world, with nothing hidden (Ch 7) +**Surrogate advantage** — importance-weighted advantage measuring new-policy performance on old-policy data (Ch 15) +**Surrogate objective** — a local approximation of `J(pi_theta)` that is safe to maximize (Ch 8, Ch 16) +**Target network** — a time-lagged copy of a network used to compute stable Bellman targets (Ch 17) +**Target policy smoothing** — clipped noise added to the target action to prevent exploiting sharp Q peaks (Ch 18) +**Trajectory (tau)** — a sequence of states and actions; also called episode or rollout (Ch 7) +**Trust region** — a KL-divergence bound on how far the new policy may move from the old (Ch 15) +**Value function** — `V^pi(s)`, expected return from s following pi (Ch 7) diff --git a/engineering/spinning-up-deep-rl/skills/spinning-up-deep-rl/patterns.md b/engineering/spinning-up-deep-rl/skills/spinning-up-deep-rl/patterns.md new file mode 100644 index 00000000..c30f890a --- /dev/null +++ b/engineering/spinning-up-deep-rl/skills/spinning-up-deep-rl/patterns.md @@ -0,0 +1,131 @@ +# Patterns + +Concrete techniques from Spinning Up, each with when to use it, how it works, and what it costs. + +## Reward-to-Go Weighting +**When to use** — always, over full-trajectory-return weighting. It is a two-line change. +**How** — weight each `grad log pi(a_t|s_t)` by `sum_{t'=t}^T r_t'` instead of `R(tau)`. +**Trade-offs** — strictly reduces variance at no cost. The dropped terms had zero mean and +nonzero variance, so removing them removes pure noise and cuts the sample trajectories needed. +(Ch 9) + +## Value Function Baseline +**When to use** — any policy gradient method. Standard in VPG, TRPO, PPO and A2C. +**How** — subtract `V_phi(s_t)` from the weight; fit `V_phi` by minimizing +`E[(V_phi(s_t) - Rhat_t)^2]` with one or more gradient steps per epoch, starting from the +previous parameters, so it tracks the most recent policy. +**Trade-offs** — provably unbiased by EGLP; reduces variance and gives faster, more stable +learning. Costs a second network and its fitting budget. Under-fitting it is one of the most +drastic performance failures in policy gradients (Exercise 2.1). (Ch 9, Ch 12) + +## Trust Region via KL Constraint +**When to use** — when a single bad policy gradient step can collapse performance, and you +can afford second-order machinery. +**How** — maximize surrogate advantage subject to `Dbar_KL <= delta`; Taylor-expand, solve +analytically via Lagrangian duality, then backtracking line search until the exact KL +constraint holds and surrogate advantage is positive. Use conjugate gradient for `H^{-1}g`. +**Trade-offs** — buys monotone-ish, fast improvement and larger safe steps. Costs complexity, +Hessian-vector products, and (in Spinning Up) no PyTorch implementation. (Ch 15) + +## Clipped Surrogate Objective +**When to use** — as the first-order replacement for a trust region. The default modern choice. +**How** — `L = min(ratio * A, g(eps, A))` where `g` caps at `(1+eps)A` for positive advantage +and `(1-eps)A` for negative. Take multiple minibatch SGD steps per batch. +**Trade-offs** — much simpler than TRPO and empirically at least as good, but it removes the +*incentive* to move far rather than guaranteeing you do not. Pair it with KL early stopping. +(Ch 16) + +## KL Early Stopping +**When to use** — alongside clipping, whenever you take many gradient steps per batch. +**How** — if the mean KL-divergence of the new policy from the old exceeds a threshold, stop +taking gradient steps for this batch. +**Trade-offs** — one extra measurement per step; catches the drift the clip alone allows. +Other PPO implementations use different tricks here. (Ch 16) + +## Experience Replay Buffer +**When to use** — every off-policy value-learning algorithm. +**How** — store `(s,a,r,s',d)` tuples; sample minibatches uniformly for MSBE minimization. +Licensed by the fact that the Bellman equation is indifferent to how transitions were collected. +**Trade-offs** — wrong in both directions. Too small and you overfit to the most recent data +and things break; too large and learning slows. It needs tuning. (Ch 17) + +## Target Networks +**When to use** — any MSBE minimization, because the target otherwise depends on the +parameters being trained. +**How** — keep a lagged copy. DQN-style: copy every fixed number of steps. DDPG-style: +polyak average once per main update, `phi_targ <- rho*phi_targ + (1-rho)*phi`, `rho` near 1. +**Trade-offs** — stabilizes learning; adds a network and a hyperparameter, and slows the +propagation of new information into the target. (Ch 17) + +## Amortizing a Continuous Argmax into a Policy +**When to use** — an inner `max` over a continuous variable inside a loop you run constantly. +**How** — assume differentiability with respect to that variable, learn `mu_theta(s)` by +gradient ascent on `Q_phi(s, mu_theta(s))`, and substitute `max_a Q(s,a) ~= Q(s, mu(s))`. +**Trade-offs** — turns an intractable per-step optimization into a forward pass. Costs an +approximation whose errors the policy will actively exploit. (Ch 17) + +## Clipped Double-Q +**When to use** — whenever a learned Q-function is being maximized over by a policy, i.e. +the whole DDPG family. +**How** — learn two Q-functions, use `min` of the two target values as the shared regression +target for both. +**Trade-offs** — fends off overestimation, which is DDPG's dominant failure. Costs a second +critic and introduces some underestimation bias. TD3 uses `Q_phi_1` alone in the policy loss; +SAC uses the `min` there too. (Ch 18, Ch 19) + +## Delayed Policy Updates +**When to use** — when policy updates destabilize the Bellman target. +**How** — update the policy and all target networks once per `policy_delay` critic updates; +the paper recommends 2. +**Trade-offs** — damps volatility; slows policy improvement per environment step. (Ch 18) + +## Target Policy Smoothing +**When to use** — deterministic policies over continuous actions. +**How** — `a'(s') = clip(mu_targ(s') + clip(eps, -c, c), a_Low, a_High)`, `eps ~ N(0, sigma)`. +**Trade-offs** — regularizes Q along the action dimension so a spurious sharp peak cannot be +exploited. A stochastic policy (SAC) gets a similar effect for free. (Ch 18, Ch 19) + +## Entropy Regularization +**When to use** — when premature convergence to a bad local optimum is the risk, or when you +want an explicit explore-exploit dial. +**How** — add `alpha * H(pi(.|s_t))` to the reward at each timestep; propagate the term into +the value functions and the Bellman target. +**Trade-offs** — higher `alpha` means more exploration and faster later learning; the right +value is environment-specific and needs careful tuning. The entropy-constrained variant that +adapts `alpha` is generally preferred by practitioners over the fixed one. (Ch 19) + +## Reparameterization Trick for Stochastic Policies +**When to use** — differentiating through an expectation whose distribution depends on the +parameters. +**How** — sample by a deterministic function of state, parameters and independent noise: +`a~ = tanh(mu_theta(s) + sigma_theta(s) * xi)`, `xi ~ N(0,I)`, converting the expectation over +actions into an expectation over noise. +**Trade-offs** — makes the policy differentiable end to end. The `tanh` squash bounds actions +but changes the distribution; log-probabilities are still closed-form. (Ch 19) + +## Uniform Random Warm-Up (start_steps) +**When to use** — every off-policy algorithm here, at the beginning of training. +**How** — for a fixed number of steps take actions sampled uniformly over valid actions, then +switch to normal exploration. +**Trade-offs** — fills the replay buffer with diverse data before the policy can bias it. +Wasted steps if the environment is expensive. (Ch 17, Ch 18, Ch 19) + +## Multi-Seed Reporting with Variance Bands +**When to use** — every reported RL result. +**How** — at least 3 seeds, 10 or more to be thorough; plot the mean as a solid line and the +standard deviation as a shaded band; smooth for display only. +**Trade-offs** — multiplies compute by the seed count. Without it, deep RL's seed sensitivity +means two seed groups can look like different distributions entirely. (Ch 10, Ch 13) + +## Precommitted Final Runs +**When to use** — before reporting any comparison. +**How** — use the tuning stage to form hypotheses, then launch fresh final experiments for +every method compared and commit in advance to reporting whatever comes out. +**Trade-offs** — a weak form of preregistration; costs one more full experiment round and +removes the largest source of accidental self-deception. (Ch 10) + +## Per-Claim Ablation +**When to use** — any method with more than one design decision, which is all of them. +**How** — swap out or remove each design element separately and measure. +**Trade-offs** — turns one bundled claim into several separately-confident ones. Costs one +experiment per element. Also the fastest way to learn which parts you can delete. (Ch 10)