openpetswithchatandmcp/FEATURES_OUR_CHANGES.md

FamiliarOS — Our Changes (Isolated Feature List)

This document lists only the features, UI surfaces, and capabilities added by our working fork. It is intentionally separate from the upstream README.md and FEATURES.md so the original repo maintainer can choose whether to adopt these changes wholesale, partially, or not at all.

Floating Chat Surface

• Double-click the familiar to open a frameless, always-on-top floating prompt window.
• Compact prompt-first UX at a minimal 96px height; expanded editor/history mode at 280px.
• Prompt bar sized for 2–3 lines of text.
• Resizable with a corner grip; size is remembered per mode (compact/expanded).
• Theme-aware (system / light / dark) shared with Control Center.
• Inline error presentation and keyboard shortcuts (Enter sends, Ctrl+Enter newline, Escape closes).

Chat History List UI

• Conversation history renders as a styled list of individual entries, not a raw textarea.
• Per-entry role badges: You, Familiar, System, Error.
• Color-coded entry backgrounds per role.
• Timestamps shown per entry.
• Scrollable with automatic scroll-to-bottom on new messages.
• Persists to disk across app restarts (familiaros-chat-history.json).

OpenAPI BYOK Chat

• Main-process OpenAPI-compatible chat service.
• Supports both responses and chat/completions transports with automatic route selection.
• Secure credential storage via Electron safeStorage with plain fallback.
• Model selection and endpoint override in Settings.
• Built-in presets: OpenAI, OpenRouter, Azure template, LiteLLM, vLLM, Moonshot/Kimi, custom/local templates.
• Localhost-only allowance for plain http.

Familiar Character & Base Instructions

• User-editable system-prompt style familiar character field in General Settings.
• Character prompt persists in app state and applies to future assistant replies.
• Base instructions toggle to include/exclude the default FamiliarOS behavior instructions in every chat.

Custom Familiar Name

• Settings → General includes a user-editable Familiar name field for the default Familiar.
• The name persists in app state (familiaros-state.json) and is applied through validatePreferencePatch.
• The chosen name appears in the tray tooltip and tray context menu.
• Falls back to the Familiar's display name or "Built-in Familiar" when no custom name is set.

Familiar Text-to-Speech (Phase 2)

• Settings > Text-to-Speech panel for the familiar and floating chat.
• System voice via the OS speech engine with voice matching and 0.5×–2.0× speed control.
• Cloud providers: OpenAI TTS, ElevenLabs, OpenAI-compatible (OpenRouter / LiteLLM / WaveSpeedAI / custom).
• Local provider: Piper for offline speech.
• Credentials stored with Electron safeStorage (plain fallback).
• Per-provider model, voice, speed, endpoint preset, and custom endpoint.
• Dynamic voice list fetching for ElevenLabs; static lists for OpenAI.
• Test voice + Stop buttons in Settings.
• Speech skips quiet hours, validates returned audio, and enforces HTTPS / localhost rules for endpoints.
• Plugin SDK voice:speak permission lets plugins speak through the same pipeline.

Memory System

• Local-first durable FamiliarOS memory store.
• Automatic memory capture from prompts (remember that..., /remember, /memorize, /note, don't forget..., my name is..., my favorite..., I prefer..., I like/love/dislike/hate...).
• Specific memory kinds (identity, preference) take precedence over generic notes when the prompt matches both.
• Relevance-based retrieval injected into future chat system prompts.
• Memory kinds: preference, identity, fact, note.
• Importance weighting and tags.
• In-app memory viewer in Settings with search, inline edit, and delete with confirmation.
• Human-inspectable markdown mirror files on disk.
• Chat history memory: cross-conversation message search that injects relevant past conversation excerpts into the system prompt.
• Separate databases for explicit memories (familiaros-memory.json) and chat history (familiaros-chat-history.json).

MCP Tool Servers (Built-in Chat)

• Internal stdio MCP client in the desktop main process (mcp-chat-client.ts).
• Top-level MCP Tool Servers panel in Control Center → Integrations.
• Multi-select tool activation for the built-in floating chat, independent of external agents.
• Tiered tool browser: Starter, Terminal & Systems, Advanced.
• Save & Activate / Deactivate All controls.
• ReAct-style tool-calling loop (max 5 iterations) with responses/chat-completions integration.
• Supported tools: filesystem, terminal/shell, memory, fetch-web, sequential-thinking, playwright, git, github, docker, sqlite.

FamiliarOS MCP Server (External-Agent Bridge)

• Central FamiliarOS MCP Server panel in Control Center → Integrations.
• Command mode selector (published / bundled / local) in one place.
• Node path override in the central panel.
• Familiar routing selector in the central panel.
• One-click Test Server health check.
• One-click Copy MCP JSON for external hosts.

MCP Toolkit Guide Integration

• MCP Toolkit Guide panel inside Control Center → Integrations.
• Hand-picked reference surface, not an automatic installer.
• Each entry explains what the tool does, what permissions it needs, and how to set it up manually or as a persistent bundle in supported hosts.
• Multi-select chips by tier (Starter, Terminal & Systems, Advanced).
• Copy-paste snippets and docs links for deliberate adoption.
• Removed redundant Curated Stack section in favor of direct tiered selection.

Knowledge Store

• Local-first file + memory knowledge repository added to the desktop app.
• Store files from Settings > Knowledge Store or directly from floating-chat attachments.
• Text extraction from UTF-8 text-like files; MIME inference from extension.
• Indexed metadata with file count cap (100) and per-file size cap (5 MB).
• Relevance search across file names and extracted text; results injected into OpenAPI chat instructions.
• Combined search across stored files and FamiliarOS memories.
• Dedicated Settings tab for upload, search, list, and delete.
• Toolbar button in the floating prompt window stores attachments into the Knowledge Store.
• Electron-free core (knowledge-store-core.ts) + unit tests (knowledge-store.test.ts).

Familiar Scale & Sizing

• Settings slider for familiar scale from 0.16x up to 10x, large enough to fill the screen.
• Familiar window dynamically resizes to fit larger scales without clipping.
• Speech bubbles grow with the familiar and larger window sizes; long messages scroll instead of cutting off.

Multi-Screen Support

• Familiar positioning uses screen.getDisplayNearestPoint() instead of always clamping to the primary display.
• clampToNearestWorkArea() keeps the familiar within the work area of the display it is currently on.

Multi-Conversation Chat

• Per-conversation message history instead of a single global transcript.
• Conversation list UI in the prompt window History button.
• Auto-generated conversation titles from the first user message.
• Create, switch, and delete conversations from the floating chat window.
• Editor button shows the current conversation's message list (unchanged).
• New Chat button creates a fresh conversation (old conversations are preserved).
• Conversation metadata stored in familiaros-chat-conversations.json.

Settings & Control Center

• Memory viewer section in Settings.
• MCP Tool Servers panel in Integrations for built-in chat tool activation.
• FamiliarOS MCP Server panel in Integrations for external-agent bridge configuration.
• OpenAPI chat settings: endpoint, model, credential, theme, base instructions toggle.
• Moonshot/Kimi API endpoint preset.

Virtual Familiar Status HUD

• Bundled familiaros.virtual-familiar plugin tracks hunger, energy, happiness, and affection.
• Status values are shown as read-only labels inside the right-click familiar context menu (Virtual Familiar ▸ Mood, Food, Energy, Play, Bond, Level), so they never block familiar clicks or double-click chat.
• Single-click the familiar to familiar it (increases bond and happiness).
• Feed / play / familiar / nap commands live in the familiar context menu.
• Low-stat nudges continue to work in the background.

Build & Packaging

• Sharp Windows native binary packaging resolved via asarUnpack + post-build copy.
• prompt-window-preload.cjs added for the floating chat renderer bridge.
• Additional preload scripts and assets packaged for the new surfaces.

Bug Fixes

• Floating-chat memory capture/retrieval is now more resilient: capture is logged and errors are caught instead of breaking the send path.
• Long speech-bubble text no longer clips; bubble bodies scroll when content exceeds the bubble max-height.