skillhub/cli/test/e2e

SkillHub CLI End-to-End Testing Guide

Where should my new test go? Use this decision tree.

Test Placement Decision Tree

What are you testing?
|
+-- Browser behavior on CLI OAuth pages (/cli/auth/*)
|   --> web/e2e/auto-generated/cli-auth/     Playwright, auto-generated by e2e-automation pipeline
|
+-- CLI subprocess vs real backend (index latency, version assignment, server rejection)
|   --> cli/test/e2e/                        THIS DIRECTORY. Bun test, real backend, skip when no token
|
+-- CLI action produces state that Web must reflect (or vice versa)
|   --> web/e2e/cross-stack-*.spec.ts        Playwright + execFileSync('bun', [...])
|
+-- Single command logic, edge cases, concurrency, resilience (fake-registry is enough)
|   --> cli/test/integration/                Bun test + fake-registry
|
+-- Pure function / class logic, no subprocess
    --> cli/test/unit/

Quick reference

Track	Dir	Framework	Backend	Skip condition
A (Web auto-gen)	web/e2e/auto-generated/cli-auth/	Playwright	dev server (localhost:3000)	never (dev server required)
B (CLI vs real backend)	cli/test/e2e/	Bun test	real SkillHub backend	SKILLHUB_E2E_REGISTRY or SKILLHUB_E2E_TOKEN missing
C (Cross-stack)	web/e2e/cross-stack-*.spec.ts	Playwright	real backend + dev server	SKILLHUB_E2E_TOKEN missing or CLI source absent
Integration	cli/test/integration/	Bun test	fake-registry (in-process)	never
Unit	cli/test/unit/	Bun test	none	never

---

This Directory (Track B)

What it tests

CLI subprocess behavior that only a real backend can validate -- things fake-registry cannot faithfully reproduce:

• Search index latency after publish (async indexing, not instant)
• Server-assigned version strings and version pinning on install
• Server-side validation rejection (missing name, overlong slug)
• Auth contract (real 401 on invalid token, not a mock)

When it skips

All suites use describe.skipIf(!live || !reachable). The skip logic lives in helpers/live-registry.ts:

1. getLiveRegistry() reads SKILLHUB_E2E_REGISTRY + SKILLHUB_E2E_TOKEN from env. Returns null if either is missing.
2. isLiveRegistryReachable(reg) probes GET /api/cli/v1/auth/whoami with a 3s timeout. Returns false on non-200 or network error.

If either check fails, every describe.skipIf block skips silently. This is intentional -- developers without a running backend still get green CI from integration and unit tests.

Current case inventory

File	Cases	What they pin
lifecycle.test.ts	E1, E5	publish -> search (exact slug) -> install -> content match -> remote remove -> search miss; search by keyword substring
upgrade.test.ts	E2	publish twice -> version bumps -> install latest gets v2 -> install --version=v1 pins to v1
errors.test.ts	E3, E4, E6, E7	install nonexistent slug -> non-zero + meaningful error; whoami with bad token -> EXIT.auth (2); publish missing name field -> rejected; publish overlong name -> rejected

How it differs from cli/test/integration/

Dimension	Integration	This directory (Track B e2e)
Backend	fake-registry (in-process HTTP server)	Real SkillHub backend
What it validates	Command logic, flag parsing, error formatting, inventory mutations	Server contract: indexing, versioning, auth, validation
Isolation	Each test gets its own fake-registry port + tmpdir	Each test uses uniqueSlug() + afterEach cleanup via deleteRemote()
CI availability	Always runs (no external deps)	Skips unless SKILLHUB_E2E_* secrets are configured

---

Adding a New Test

Checklist:

1. Use runCli() from ../helpers/run-cli.ts. Do not call Bun.spawn directly.
2. Isolate home directory with createTempHome() from ../helpers/temp-env.ts. Pass { HOME: env.home, USERPROFILE: env.home } as env to prevent leaking host credentials.
3. Use uniqueSlug() for any published skill name. Concurrent or repeated runs must not collide.
4. Clean up in afterEach. Track published slugs in an array; pop and deleteRemote() each one. deleteRemote is best-effort and never throws.
5. Use eventually() for assertions that depend on async indexing. Default: 5s timeout, 250ms interval. Bump to 8-10s for search index assertions.
6. Set a generous test timeout (15-30s) since real backend round-trips add latency.
7. Prefer --json flag on CLI calls and assert against parsed JSON fields rather than human-readable text. Keeps tests stable across i18n or formatting changes.

---

Environment Variables

SKILLHUB_E2E_REGISTRY

The base URL of a running SkillHub backend. Example: http://localhost:3000.

This is the same server the web dev stack uses. If you already have docker-compose up running for web development, point here.

SKILLHUB_E2E_TOKEN

A bearer token with publish and delete permissions. This must be an operator/API token, not a session cookie. Why:

• Session cookies go through SessionAuthenticationFilter; API tokens go through ApiTokenAuthenticationFilter. They have different visibility scoping rules.
• Track B and Track C both use one well-known operator token to keep tests deterministic.

How to obtain one:

1. Log in to the SkillHub web dashboard.
2. Navigate to Settings > API Tokens (or /dashboard/settings/tokens).
3. Create a token with a descriptive label like e2e-cli-tests.
4. Copy the sk_... value.

Local setup

Create a .env file (not committed) in the repo root or export in your shell:

export SKILLHUB_E2E_REGISTRY=http://localhost:3000
export SKILLHUB_E2E_TOKEN=sk_your_token_here

CI status (as of 2026-05-07)

Workflow	File	Runs Track B?	Runs Track C?
pr-cli.yml	.github/workflows/pr-cli.yml	Skips (no SKILLHUB_E2E_* secrets configured)	N/A (not in scope)
pr-e2e.yml	.github/workflows/pr-e2e.yml	N/A (doesn't watch cli/)	Skips (no SKILLHUB_E2E_TOKEN secret configured)

To enable in CI: add SKILLHUB_E2E_REGISTRY and SKILLHUB_E2E_TOKEN as repository secrets in GitHub, then reference them in the workflow env: block. Track B tests will automatically un-skip once the env vars are present.

---

Running Locally

# Without token -- confirms test files parse and skip cleanly
cd cli
bun test test/e2e/

# With token -- actually hits the real backend
export SKILLHUB_E2E_REGISTRY=http://localhost:3000
export SKILLHUB_E2E_TOKEN=sk_...
bun test test/e2e/

# Single file
bun test test/e2e/lifecycle.test.ts

# With verbose output
bun test test/e2e/ --verbose

Test reports are generated by scripts/junit-to-html.ts and written to reports/.

---

Deferred Work

See cli/test/BACKLOG.md for items deferred from this directory:

• TODO-001: Dual-user namespace-only / private isolation (authenticated-but-not-a-member case). Needs a second operator token fixture. Deferred because X4/X5 in cross-stack already pin the anonymous boundary; marginal value is real but fixture cost is ~0.5d.

---

Related Docs

• Track A details: web/e2e/auto-generated/README.md
• Pipeline that generates Track A: e2e-automation/README.md
• Track C source: web/e2e/cross-stack-cli-publish.spec.ts (see file header for X1-X7 descriptions)
• Integration tests: cli/test/integration/ (one file per CLI command)
• Test backlog: cli/test/BACKLOG.md