271 lines
16 KiB
Markdown
271 lines
16 KiB
Markdown
# FamiliarOS Plugin Platform (SDK v3 / SuperPlugins)
|
||
|
||
This is the reference for the current plugin platform. The forward-looking
|
||
spec that drove this implementation is `docs/superplugins.md`; this document
|
||
describes what is actually built and how to work with it.
|
||
|
||
## Architecture
|
||
|
||
```text
|
||
plugin package (manifest + entry + assets + panels)
|
||
-> plugin-service.ts install/seed/local-load, state, config
|
||
-> plugin-runtime.ts per-plugin lifecycle, declarative timers
|
||
-> plugin-js-host.ts sandboxed BrowserWindow per JS plugin + IPC dispatch table
|
||
-> plugin-sdk-preload.cjs builds the plugin-facing ctx (handles, subscriptions)
|
||
-> plugin-sdk-bridge.ts validation, permissions, quotas (the security boundary)
|
||
-> plugin-host-capabilities.ts Electron side effects (one instance, injected at startup)
|
||
bubbles -> plugin-bubble-arbiter.ts -> default-familiar-controller / plugin-familiar-registry -> familiar-window.ts
|
||
audio/tts -> familiar window renderer (WebAudio recipes, data-URL playback, speechSynthesis)
|
||
events -> plugin-events-source.ts (familiar windows, powerMonitor, screen, pollers)
|
||
familiars -> plugin-familiar-registry.ts + familiar-motion-engine.ts (spawn, moveTo, followCursor, physics, onTick)
|
||
panels -> plugin-panels.ts (+ panel-preload.cjs)
|
||
ai -> plugin-ai-gateway.ts (Anthropic / OpenAI / Ollama)
|
||
secrets -> plugin-secrets.ts (safeStorage-encrypted, per plugin)
|
||
auth -> plugin-oauth.ts (PKCE loopback, system browser)
|
||
voice -> plugin-voice.ts (TTS via renderer; one-shot STT via capture window + provider)
|
||
toast -> plugin-toast.ts; notify -> Electron Notification
|
||
settings -> plugin-platform-settings.ts (sound/AI-speech/voice/mic toggles + quiet hours + AI provider)
|
||
```
|
||
|
||
The single governing rule: **plugins describe; the host renders.** Bubbles are
|
||
descriptors validated in the bridge and rendered by `familiar-window.ts`. Plugin
|
||
HTML runs only inside the sandboxed *panel* window (`ui:panel`), never in a familiar
|
||
window.
|
||
|
||
Bridge naming note: the desktop sandbox now exposes FamiliarOS-first globals
|
||
(`window.familiarOSPanel`, `window.familiarOSCommandForm`,
|
||
`window.familiarOSPromptWindow`, `globalThis.__familiarOSSdk`,
|
||
`globalThis.__familiarOSRunCallback`) while retaining the historical
|
||
OpenPets names as compatibility aliases during the rebrand transition.
|
||
|
||
## Manifest
|
||
|
||
`familiaros.plugin.json`, validated by `apps/desktop/src/plugin-manifest.ts`.
|
||
A JSON Schema ships with the CLI (`packages/cli/schemas/familiaros.plugin.schema.json`);
|
||
manifests may carry `$schema` for editor validation.
|
||
|
||
- `manifestVersion: 1` — declarative timer plugins (legacy).
|
||
- `manifestVersion: 2` — JavaScript SDK v2 surface (legacy; keeps working unchanged).
|
||
- `manifestVersion: 3` — SDK v3. Requires `sdkVersion: "3.x.y"`. Adds:
|
||
- `assets`: `{ icons | images | svgs | sprites | sounds: { name: relativePath } }` —
|
||
up to 32 entries per kind. Validated at install: path containment, format,
|
||
per-kind size caps (`pluginAssetMaxBytes`), and SVG sanitization
|
||
(`plugin-assets.ts` strips script/foreignObject/event handlers/external hrefs).
|
||
- `panels`: `{ name: relative .html path }` (max 8). Panel HTML gets a strict
|
||
CSP injected at install.
|
||
- New config field types `date`, `sound`, and `secret` (masked input; no defaults allowed).
|
||
|
||
Minimum unblock note: Control Center `sound` config fields now support importing
|
||
`.ogg`, `.mp3`, and `.wav` files through the host picker. Imported sounds are
|
||
stored as opaque `{ kind: "user-sound", id, name }` refs; raw filesystem paths
|
||
are rejected by config validation. Near-term debt remains to split
|
||
`plugin-sdk-bridge.ts` by namespace, centralize preload/host route contracts,
|
||
extract the user-sound store from host capabilities, and add parity tests for
|
||
renderer preload routes.
|
||
|
||
## Permissions (v3)
|
||
|
||
Existing: `familiar:speak`, `familiar:reaction`, `familiar:move`, `schedule`, `storage`,
|
||
`status`, `commands`, `network` (+ declarative-only `timer`).
|
||
|
||
New in v3: `familiar:interact` (bubble buttons/inputs), `familiar:pin` (pinned slot),
|
||
`familiar:animate` (custom sprites/scale), `familiar:speak:dynamic` (AI-generated speech),
|
||
`familiar:drop` (drag-and-drop onto the familiar), `familiars:read`, `familiars:manage` (spawn/close),
|
||
`audio`, `events`, `ui:toast`, `ui:panel`, `notify`, `bus`, `ai`, `secrets`,
|
||
`voice:speak`, `voice:listen`, `auth`, `files`, `system:openExternal`,
|
||
`system:metrics`, `clipboard`, `network:write`.
|
||
|
||
Flagged **sensitive** (louder consent, gated by global settings, default off):
|
||
`voice:listen`, `clipboard`, `familiar:speak:dynamic`
|
||
(`sensitivePluginPermissions` in `plugin-manifest.ts`).
|
||
|
||
Trust model: declared permissions + user approval at install + catalog review.
|
||
No signing tier (deliberate — see `docs/superplugins.md` §15).
|
||
|
||
## The SDK surface
|
||
|
||
Types: `packages/sdk/src/index.ts` (`@familiaros/plugin-sdk`, v3). The
|
||
namespaces on `ctx`: `familiars`, `familiar` (alias of `familiars.default`), `ui` (bubbles,
|
||
alert, toast, panel, dynamic menu), `audio`, `events`, `assets`, `bus`, `schedule`
|
||
(`once`/`every`/`daily`/`cron`/`at`/`list`), `storage` (now with `keys` +
|
||
`subscribe`, ~5 MB quota), `config`, `net` (`fetch` with non-GET +
|
||
`stream`), `notify`, `ai`, `secrets`, `voice`, `auth`, `files`, `system`,
|
||
`commands`, `status`, `http` (v2 GET-only alias), `log`.
|
||
|
||
Hard lines kept regardless of permissions:
|
||
|
||
- The render rule above (no raw markup into familiar windows).
|
||
- The privacy line (§3.1): no keystrokes, no screen contents, no other apps'
|
||
|
||
- `ctx.ui.alert(...)` is the must-not-miss delivery helper: it renders a sticky,
|
||
high-priority familiar bubble and can optionally request `sound`, `notify`, actions,
|
||
`dismissOn`, an `indicator`, and rich bubble content (`text`, limited
|
||
`markdown`, `icon`, `svg`, `image`, `tone`). `indicator` renders the top header
|
||
row used by familiar status messages, but is alert-owned instead of a familiar reaction: it
|
||
accepts a named host icon or a manifest-declared asset via `ctx.assets.icon(...)`
|
||
/ `ctx.assets.svg(...)` / `ctx.assets.image(...)`, plus safe `color`,
|
||
`background`/`backgroundColor`, and `borderColor` values. Raw SVG strings are
|
||
not accepted at runtime; bundle SVGs in `assets` so the host can validate and
|
||
sanitize them at install. Bubble body media (`icon`, `svg`, `image`) is
|
||
icon-only and cannot be combined with `text`/`markdown`; use `indicator` for
|
||
icon + message alerts. Alerts require `familiar:speak`; `familiar:interact` is only
|
||
needed for actions/input, `audio` only when `sound` is set, and `notify` only
|
||
when `notify` is set. The returned handle behaves like a bubble handle and adds
|
||
`acknowledge()`.
|
||
- Config schemas may use `type: "sound"` for host-managed plugin sound
|
||
preferences. The saved value is a named host sound, an opaque user sound ref,
|
||
or empty; plugins never receive raw filesystem paths.
|
||
- The privacy line (§3.1): no keystrokes, no screen contents, no other apps'
|
||
window titles, no ambient clipboard/microphone/filesystem. Clipboard read is
|
||
allowed only *inside a user-invoked command handler*; STT is one-shot
|
||
push-to-talk behind a default-off toggle; drops fire only on explicit drags.
|
||
- Network: HTTPS-only, manifest-declared + user-approved exact hosts, manual
|
||
redirects, response caps, DNS/private-IP SSRF guard (`assertPublicHost`).
|
||
|
||
Quotas live in `pluginSdkQuotas` (`plugin-sdk-bridge.ts`).
|
||
|
||
Reactions require `familiar:reaction`: `ctx.familiar.react(reaction)` preserves the built-in
|
||
reaction/status bubble, while `ctx.familiar.react(reaction, { showMessage: false })`
|
||
plays the animation without generating that default text.
|
||
|
||
## Plugin i18n
|
||
|
||
A plugin ships its translations as `locales/<locale>.json` — one file per
|
||
supported locale, the same convention as the host catalog: a flat map of dotted
|
||
keys to strings, with `{var}` interpolation. `locales/en.json` is the source and
|
||
the fallback; missing locales (or missing keys within a locale) fall back to
|
||
`en`, then to the raw key. The host packages and loads any present `locales/`;
|
||
no file is required.
|
||
|
||
Two ways to use those keys:
|
||
|
||
- **`$t:key` references in manifest static fields** — wherever the host renders
|
||
a plugin-authored string at display time: `name`, `description`, `configSchema`
|
||
labels/descriptions/option labels, command titles/descriptions, and dynamic
|
||
menu item titles. Write the value as `"$t:plugin.name"`; the host resolves it
|
||
against the plugin's catalog for the active locale (→ plugin `en` → raw key) at
|
||
display time, so labels re-render translated when the user switches language.
|
||
- **`ctx.t(key, vars?)` + `ctx.locale`** — for strings the plugin composes at
|
||
runtime (bubble / notify / status bodies with interpolation). `ctx.t` reads the
|
||
active locale live and interpolates `{var}` placeholders; `ctx.locale` is the
|
||
current locale string. Example: `ctx.t("reminder.due", { message })`.
|
||
|
||
Keep placeholders intact across locales and leave brand names untranslated. The
|
||
reference implementations include `familiaros.reminders` ("Quick Reminders") and
|
||
`familiaros.launch-buddy` ("Launch Buddy"), with the reminder flow mirrored by the
|
||
CLI `reminder` template.
|
||
|
||
## Bubbles & the arbiter
|
||
|
||
`ctx.ui.bubble(spec)` / `familiar.speak(spec)` accept a string or a descriptor
|
||
(text, limited markdown, icon/svg/image refs, tone, accent token, duration,
|
||
sticky, pin, dismissOn, priority, actions, input, hud) and return a live handle
|
||
(`update`, `dismiss`, `pin`, `unpin`, `onAction`, `onSubmit`, `onDismiss`).
|
||
|
||
A **pinned mini HUD bubble** can be rendered using the `hud` descriptor (requires `pin: true`). The `hud` property takes an `items` array (1–4 items), where each item has `icon` (named host icon or asset ref), `value` (0–100), optional `label`, and optional `tone` ("amber", "blue", "green", "pink", "slate", "red"). When `hud` is present, it must not be combined with text, markdown, body media, or indicator. Pinned bubbles with `hud` render as a compact, polished 2x2 grid with CSS progress bars, avoiding emoji alignment issues.
|
||
|
||
`plugin-bubble-arbiter.ts` (one per familiar surface) arbitrates: priority queue,
|
||
do-not-interrupt for sticky/urgent, coalescing of identical back-to-back
|
||
messages, and a single **pinned slot** above the transient slot with
|
||
priority-aware replace semantics. Non-dynamic transient text goes through the
|
||
static content filter and stays single-line; pinned text may use a few safe lines
|
||
for compact status HUDs. `dynamic: true` content needs `familiar:speak:dynamic` plus
|
||
the global toggle and gets the relaxed screen (2,000 chars, secret redaction).
|
||
|
||
## Multi-familiar & liveness
|
||
|
||
`ctx.familiars.spawn({ petId })` opens an ephemeral window for an installed familiar
|
||
(max 4 per plugin), addressable via handles. `onTick` is the host-driven brain
|
||
loop (~10 fps, paused while hidden/dragging); `getState` gives self-perception;
|
||
`moveTo`/`followCursor`/`physics` run in `familiar-motion-engine.ts`. Spawned familiars
|
||
are torn down with their plugin.
|
||
|
||
## Host integrations
|
||
|
||
- **AI gateway** — one user-configured provider (Settings → Plugin Platform):
|
||
Anthropic, OpenAI, or Ollama. Keys are safeStorage-encrypted and never reach
|
||
plugin code. `complete` supports tools (function calling); `stream` streams
|
||
tokens. BYO-provider plugins can use `net.stream` + `secrets` instead.
|
||
- **OAuth** — `ctx.auth.oauth` runs PKCE against a loopback listener in the
|
||
system browser; tokens persist in the plugin's secrets and are returned to
|
||
the plugin. `refresh`/`signOut` manage the stored session.
|
||
- **Files** — OS dialogs only (`pick`/`save`); reads are size-capped, one-shot
|
||
handles. Dropped files (`familiar:drop`) are readable through the same accessor.
|
||
- **System** — `info()` (platform/locale/timezone/theme/version/online) is
|
||
always available; `metrics()` (aggregate CPU/mem only) needs
|
||
`system:metrics`; `openExternal` is HTTPS-only.
|
||
- **Commands** — `ctx.commands.register(...)` accepts `icon` as either one of
|
||
the host's named icon strings (for example `"info"`, `"check"`, `"timer"`) or
|
||
a manifest-declared bundled icon reference from `ctx.assets.icon(name)`. Raw SVG
|
||
strings are rejected; put custom SVG/PNG icons under `assets.icons` so the host
|
||
can validate and sanitize them before the command reaches runtime/UI state.
|
||
- **Quiet hours** are a host primitive (Settings → Plugin Platform) gating
|
||
speech audio, plugin sound, voice, and notification sound together.
|
||
|
||
## Developer workflow
|
||
|
||
```bash
|
||
# scaffold (templates: blank | reminder | ambient | ai-chat | tamagotchi | calendar)
|
||
npx @familiaros/cli plugin new "My Plugin" --template tamagotchi
|
||
|
||
# author-time validation: manifest, config schema, permissions, declared files
|
||
npx @familiaros/cli plugin validate ./my-plugin
|
||
|
||
# deterministic tests without the app (fake clock, event injection, mocks)
|
||
npm test # runs test.js against @familiaros/plugin-sdk/testing
|
||
|
||
# live with hot reload — saving a file re-snapshots and reloads just that
|
||
# plugin, preserving enabled state, storage, and approved permissions
|
||
FAMILIAROS_DEV_PLUGIN_PATHS=$(pwd)/my-plugin pnpm dev:desktop
|
||
# or for a folder of plugins (this is what pnpm dev:desktop:plugins uses)
|
||
FAMILIAROS_DEV_PLUGIN_ROOTS=$(pwd)/plugins/official pnpm dev:desktop
|
||
```
|
||
|
||
The test kit is `@familiaros/plugin-sdk/testing` (`createTestHarness`): fake
|
||
clock (`clock.advance("90m")` drives `once`/`every`/`daily`/`cron`/`at`),
|
||
curated event injection (`emit`), bubble interaction
|
||
(`fireBubbleAction`/`fireBubbleSubmit`), command runs, permission simulation
|
||
(unapproved namespaces throw), and mocks for `net`, `ai`, `secrets`, `files`,
|
||
`auth`, `voice`, and `system`. Assertions: `expectSpoke`, `expectReacted`,
|
||
`expectScheduled`, `expectBubble` (matches descriptors, not pixels),
|
||
`expectStored`, `expectNetCall`, `expectNotified`, `expectNoErrors`.
|
||
|
||
Runtime introspection: `familiaros:plugins-inspector` IPC
|
||
(`runtime.getInspectorState(id)`) returns schedules + next runs, registered
|
||
commands/menu items, active bubbles/panels, subscription counts, quota
|
||
counters, and the last error.
|
||
|
||
## Publishing
|
||
|
||
Unchanged flow: `pnpm plugins:check` validates and packages `plugins/official`
|
||
into ZIPs + catalog (dry-run); `plugins:publish` uploads. v3 ZIPs may contain
|
||
the manifest, the entry, and every declared asset/panel file — nothing else.
|
||
Catalog min/max-FamiliarOS-version metadata gates what loads where; v2 plugins
|
||
keep loading through the same runtime (the v3 context is a superset).
|
||
|
||
## Drift guards & CI
|
||
|
||
- `apps/desktop/src/check-plugin-sdk-conformance.ts` — compile-time guard that
|
||
the bridge surface and permission union match the published SDK types.
|
||
- `packages/sdk/src/check-plugin-sdk.ts` — runtime contract test of the mock
|
||
context (used by `pnpm --filter @familiaros/plugin-sdk test`).
|
||
- `apps/desktop/tests/plugin-bridge-fuzz.test.ts` — property/fuzz tests over
|
||
the bridge validators (cron, markdown, dynamic-text redaction, SVG/panel
|
||
sanitizers, private-IP guard, form values, arbiter invariants).
|
||
- `pnpm check` runs all of the above.
|
||
|
||
## Troubleshooting
|
||
|
||
- *Bubble never shows*: check the inspector for quota counters and the
|
||
arbiter state; sticky/urgent bubbles block lower-priority ones.
|
||
- *`Plugin permission is not approved`*: the manifest must declare it AND the
|
||
user must have approved it (re-load local plugins after permission changes).
|
||
- *Audio/voice silent*: check Settings → Plugin Platform toggles and quiet
|
||
hours.
|
||
- *`ai.*` throws*: configure a provider + key in Settings → Plugin Platform.
|
||
- *Sprite override not rendering*: sprites must be a horizontal strip with
|
||
square frames (frame size = image height); fps 1–30.
|
||
- *Renderer-visible URL schemes*: per `AGENTS.md`, any new scheme needs CSP
|
||
updates in `apps/desktop/vite.config.ts` and
|
||
`apps/desktop/src/renderer/index.html`. The v3 features deliberately reuse
|
||
`file:` (already allowed in familiar windows) and add no new schemes.
|