openpetswithchatandmcp/FEATURES.md

FamiliarOS Feature Inventory

This file is the current feature roll-up for this working fork of FamiliarOS. It combines:

• the shipped baseline described in README.md
• the chat, memory, and prompt-window work from this implementation thread
• the MCP Toolkit Guide surface added in the desktop app

Looking for an isolated view of only our additions? See:

• FEATURES_OUR_CHANGES.md
• README_OUR_CHANGES.md
• PULL_REQUEST.md

Desktop Companion Core

• Tray-first desktop companion app
• Desktop familiar that can idle, react, wave, move, and speak
• Default familiar plus installed non-default familiars
• Per-agent familiar routing when integrations request a specific installed familiar
• Manual familiar dismissal with lease-aware recovery rules
• Native right-click familiar context menu
• Always-on-top familiar window with click-through background behavior
• Drag support with mouse passthrough recovery for Windows/Linux edge cases
• Adaptive familiar scale setting (continuous slider from 0.16x to 10x in Settings)
• Reset default familiar position
• Launch at login support where the platform allows it
• Update checking against GitHub releases
• Open logs folder from the app

Familiar Packs And Gallery

• Built-in default familiar
• Installed familiar pack loading
• Local familiar import from ZIP
• Local familiar import from folder
• Codex familiar import flow
• Familiar catalog browsing
• Familiar preview thumbnails and spritesheets
• Default familiar selection
• Familiar gallery linkout

Reactions And Speech

• Explicit familiar speech bubbles
• Reaction-only decorative bubble messages
• Reaction animation mapping
• User-configurable reaction-to-animation overrides
• Longer display duration for longer messages
• Separate handling for finite reaction animations vs visible bubble lifetime
• Safer speech rules for agent-driven bubble content
• Bubble behavior that avoids showing code, logs, URLs, paths, or secrets in normal integration speech

Familiar Text-to-Speech (Phase 2)

• Settings > Text-to-Speech panel controls voice output for the familiar and floating chat.
• System voice uses the OS speech engine through the renderer (window.speechSynthesis) with optional voice-name matching and a 0.5×–2.0× speed multiplier.
• Cloud TTS providers: OpenAI TTS, ElevenLabs, and an OpenAI-compatible preset for OpenRouter, LiteLLM, WaveSpeedAI, or a custom endpoint.
• Local TTS provider: Piper (spawned as a local process) for fully offline speech.
• Provider credentials are stored with Electron safeStorage and fall back to plain local storage when encryption is unavailable (apps/desktop/src/tts-credentials.ts).
• Per-provider model, voice, speed, endpoint preset, and custom endpoint controls.
• Dynamic voice list fetching for ElevenLabs; static voice lists for OpenAI / OpenAI-compatible.
• In-app Test voice and Stop buttons to preview the configured voice.
• Assistant replies and other speech are spoken through the familiar window renderer via a shared TTS service (apps/desktop/src/tts-service.ts).
• Respects quiet-hours: speech is skipped while quiet hours are active.
• Audio returned by cloud providers is validated (MP3/WAV magic bytes) and played through a renderer <audio> element.
• Endpoint validation enforces HTTPS for cloud providers, allows http://localhost for local proxies, and blocks private IP endpoints for cloud providers.
• Input capped at 2,000 characters; clear error messages when synthesis fails.
• Plugin SDK bridge: plugins granted the voice:speak permission can ask the familiar to speak text through the same TTS pipeline.

Virtual Familiar Care

• Bundled familiaros.virtual-familiar plugin tracks hunger, energy, happiness, and affection
• Stats decay slowly while the app is running and recover when you feed, play, familiar, or nap
• Status values live in the right-click familiar context menu (Virtual Familiar ▸ Mood, Food, Energy, Play, Bond, Level); no on-familiar HUD overlay
• Single-click the familiar to familiar it (increases bond and happiness)
• Feed, play, and nap actions are also available from the familiar context menu
• Automatic nudges when a stat gets low

Custom Familiar Name

• Settings → General includes a Familiar name field for the default Familiar
• The name persists in familiaros-state.json and is applied through validatePreferencePatch
• The chosen name appears in the tray tooltip and the tray context menu
• The menu falls back to the Familiar's display name or "Built-in Familiar" when no custom name is set

Floating Chat Surface

• Double-click the familiar to open a floating prompt window
• Floating prompt window stays always on top
• Compact prompt-first mode at 96px height; expanded editor/history mode at 280px
• Prompt bar is sized for 2–3 lines of text
• Resizable corner grip with size remembered per mode
• Prompt window keeps a low desktop footprint
• Minimal top-bar controls with small symbol buttons
• Editor panel toggle
• History toggle
• New chat action
• Settings shortcut
• Close action
• Copy chat support
• Styled conversation history list with per-entry role badges and timestamps
• Scrollable message entries with distinct user/assistant/system/error theming
• Enter sends
• Ctrl+Enter or Cmd+Enter inserts newline
• Inline error presentation inside the prompt surface
• No terminal required for normal chat usage

Chat History

• Full conversation history stored and persisted to disk across app restarts
• History survives closing and reopening the floating chat window
• History stays in the prompt/editor surface rather than in the familiar bubble
• Familiar bubble only shows one assistant reply at a time
• Per-entry styling with role badges (You / Familiar / System / Error) and timestamps
• Multi-conversation support: each conversation is a separate thread with its own messages
• Conversation list accessible from the History button in the floating chat window
• Editor button shows the current conversation's message list
• Create, switch, and delete conversations from the floating chat window
• Auto-generated conversation titles from the first user message
• Old conversations are preserved when starting a new chat

OpenAPI And BYOK Chat

• Main-process OpenAPI chat service
• User-entered prompt stays out of the familiar renderer
• User-entered prompt stays out of logs unless logging is genuinely necessary elsewhere
• API key or token entry in Settings
• Secure storage with Electron safeStorage when available
• Plain local fallback when platform encryption is unavailable
• Model selection in Settings
• OpenAPI-compatible endpoint override in Settings
• Built-in OpenAI endpoint default
• BYOK endpoint support for other compatible providers or local gateways
• Endpoint normalization and validation
• Localhost-only allowance for plain http
• Automatic responses plus chat/completions route selection for broader provider compatibility
• Preset-based provider setup
• Compact single credential entry surface that adapts to the selected provider preset

Current Provider Presets

• OpenAI
• OpenRouter
• Azure template
• LiteLLM local
• vLLM local
• Custom local template
• Generic HTTPS template

Familiar Character And Personality

• System-prompt style familiar character field in General Settings
• Character prompt persists in app state
• Character prompt is applied on future assistant replies
• Base instructions toggle — optionally include or exclude the default FamiliarOS behavior instructions in every chat
• Theme setting shared between control center and floating chat window
• Auto theme mode
• Light theme mode
• Dark theme mode
• Dark theme tuned toward graphite-style surfaces instead of blue-heavy dark UI

Memory System

• Local-first persistent FamiliarOS memory store
• Main-process memory writer
• In-memory retrieval during chat
• On-disk persistence for durable memories
• Human-inspectable memory mirror files
• Relevance-based retrieval into future prompts
• Explicit memory capture patterns such as remember ...
• Preference, identity, fact, and note style memory kinds
• Importance weighting
• Tags
• Forget/delete support
• In-app memory viewer in Settings with search, edit, and delete
• Chat history memory: cross-conversation message search injects relevant past conversation excerpts into the system prompt
• Two separate databases: explicit memories (familiaros-memory.json) and chat history (familiaros-chat-history.json)

Knowledge Store

• Local-first file and memory knowledge repository for the desktop app
• Store files from Settings or directly from floating-chat attachments
• File metadata index persisted in familiaros-knowledge-store.json
• Stored file contents copied to userData/knowledge/files/ with sanitized names
• Automatic UTF-8 text extraction from text-like files (TXT, MD, JSON, code files, CSV, YAML, etc.)
• MIME type inference from file extension when not supplied
• File search by token and phrase overlap across file names and extracted text
• Relevance scoring with phrase bonus, recency weighting, and size penalty
• Relevant file context injected into OpenAPI chat system instructions
• Combined search across stored files and FamiliarOS memories
• Dedicated Knowledge Store tab in Settings with upload, search, list, and delete
• Store attached prompt-window files into the Knowledge Store via toolbar button
• File count cap (100 files) and per-file size cap (5 MB)
• Invalid-file rejection: empty files, non-Buffer data, missing names, and path-traversal names
• Electron-free core class (knowledge-store-core.ts) for deterministic Node unit tests

FamiliarOS MCP Server (External-Agent Bridge)

This MCP server exposes your Familiar to Claude Code, Cursor, OpenCode, Codex CLI, or any other MCP-capable host. It is a small, controlled bridge: external agents can make your Familiar react, speak a safe bubble, or read/write memory, but they do not get unrestricted access to your desktop.

• Local IPC discovery file and per-run token flow
• MCP server for FamiliarOS desktop control
• Explicit target-familiar lease routing
• Default-familiar fallback when a requested familiar is unavailable
• familiaros_status
• familiaros_react
• familiaros_say
• familiaros_memory_list
• familiaros_memory_search
• familiaros_memory_store
• familiaros_memory_forget
• Central FamiliarOS MCP Server panel in Control Center → Integrations
• Command mode selector (published / bundled / local) in one place
• Node path override in the central panel
• Familiar routing selector in the central panel
• One-click Test Server health check
• One-click Copy MCP JSON for external hosts

Agent Integrations

• Claude Code integration
• OpenCode integration
• Cursor integration
• Pi extension guidance
• Global setup preview/state for Claude
• Global setup preview/state for OpenCode
• Global setup preview/state for Cursor
• Managed Claude memory instructions
• Managed Claude hook install/uninstall
• OpenCode instruction file support
• OpenCode plugin support
• Cursor MCP config preview
• Cursor rules preview
• Familiar routing centralized in the FamiliarOS MCP Server panel
• Published package / bundled / local command modes centralized in the FamiliarOS MCP Server panel
• Agent-specific command path overrides (Claude command, OpenCode command)
• Node command path centralized in the FamiliarOS MCP Server panel

MCP Tool Servers (Built-in Chat Tools)

The MCP Tool Servers panel gives the built-in floating chat the ability to do real work on your local machine. It is separate from the external-agent bridge and works without any external agent installed.

• Top-level MCP Tool Servers panel in Control Center → Integrations
• Multi-select MCP tool activation for the built-in floating chat
• Works without Claude, Cursor, OpenCode, or Pi installed
• Tiered tool browser (Starter, Terminal & Systems, Advanced)
• Save & Activate / Deactivate All controls
• Tools run through an internal stdio MCP client in the main process
• Supported tools: filesystem, terminal, memory, fetch-web, sequential-thinking, playwright, git, github, docker, sqlite

MCP Toolkit Guide Surface

The control-center Integrations page includes an MCP Toolkit Guide panel. It is not an automatic installer; it is a hand-picked reference that explains what popular MCP tools do, what permissions they need, and how to set them up manually or as a persistent bundle in supported hosts. Each entry includes copy-paste snippets and guidance so you can adopt the MCP ecosystem deliberately.

Starter Stack In The Toolkit

• Filesystem
• Git
• GitHub
• Playwright
• Browser Use
• SQLite / PostgreSQL guidance
• Memory
• Fetch / Web
• Sequential Thinking
• Docker MCP Toolkit

Terminal And Systems Layer In The Toolkit

• Shell / Terminal guidance
• Process & Logs guidance
• System Info guidance
• SSH guidance
• Package Manager guidance
• Tmux guidance

Later / Advanced Layer In The Toolkit

• Kubernetes / Cloud guidance
• CI / CD guidance
• Ghidra guidance
• Binary Analysis guidance
• Network Analysis guidance

Toolkit UX

• Toolkit card in Integrations
• Explicit Manual Setup vs Persistent Full Access choice
• Command source labels clarify where tools come from (Stable release, Shipped with app, Local build)
• Install choice descriptions clarify manual vs persistent access
• Tiered MCP family browser
• Per-tool rationale
• Per-tool VectorShell-fit notes
• Per-tool permission-boundary notes
• Copyable install/config snippets where they are stable enough to recommend
• External docs opening through Electron
• Persistent host install bundles for supported hosts
• Install Now actions for supported Claude Code and Codex CLI baselines
• Installed/skipped MCP result reporting inside the toolkit UI
• Current automatic baseline: Filesystem, Playwright, Memory, Context7 / Docs, Sequential Thinking, plus Fetch / Web and Browser Use when uvx is available

Plugins

• Plugin catalog
• Installed plugin inspection
• Enable/disable plugins
• Local plugin loading
• Catalog plugin install/update/remove
• Plugin configuration schema rendering
• Plugin command execution
• Plugin status display

Dashboard And Control Center

• Dashboard overview
• Familiar count
• Plugin health summary
• Update status summary
• Reaction mix charting
• Top companion activity
• Activity counters
• Settings view
• Familiars view
• Plugins view
• Integrations view
• Persistent route handling inside the control center window

Logging And Diagnostics

• App log file
• Previous log rollover
• Redaction of token-like values
• Verbose dev logging toggle through environment
• IPC request logging
• Lease acquisition logging
• Familiar show lifecycle logging
• Renderer console logging capture

Packaging And Distribution

• Desktop packaging through Electron Builder
• Windows packaging support
• Linux AppImage packaging support
• Additional Linux targets in config
• Packaged preload scripts for control center, familiar window, prompt window, and plugin surfaces
• Packaged official plugins as extra resources
• Packaging contract checks

Safety And Boundary Rules

• Prompt text is kept out of the familiar renderer
• Automatic agent speech is decorative and best effort
• FamiliarOS speech bubbles are not meant to carry code, logs, URLs, file paths, or secrets
• Tooling guidance in the MCP Toolkit emphasizes minimal permissions
• Docker MCP Toolkit is recommended as an isolation layer when the third-party MCP ecosystem feels too uneven

Practical Summary

This fork now gives you three layers at once:

• a playful desktop familiar
• a floating always-on-top chat surface with BYOK and memory
• a curated MCP/workbench surface for building a serious developer stack around the familiar instead of only driving it from a terminal