openpetswithchatandmcp/PULL_REQUEST.md

Pull Request: Floating Chat, Memory, Knowledge Store, MCP Servers, Curated MCP Toolkit, TTS Phase 2 & Familiar Sizing

Title: feat(desktop): add floating chat, local memory, Knowledge Store, MCP Tool Servers, FamiliarOS MCP Server panel, Curated MCP Toolkit, Text-to-Speech Phase 2, expanded familiar sizing, and multi-screen support

Author: VectorShell working fork Base: upstream main at cba81b7 Status: ready for review / adopt selectively

---

Summary

This PR adds a complete in-app chat layer, a durable local memory system, interactive MCP tool activation, and several companion-quality improvements to the desktop familiar. Every change is additive and preserves existing MCP/agent integration behavior.

Motivation

• Users wanted to chat with the familiar without a terminal or external agent.
• Long-running companions benefit from remembering facts, preferences, and identity.
• A curated but practical MCP surface helps users adopt tools without unsafe one-click installs.
• Familiar sizing was limited to three presets with no continuous or interactive adjustment.

What changed

New surfaces

• Floating Prompt Window (apps/desktop/src/prompt-window.ts)

  • Frameless, always-on-top chat UI opened by double-clicking the familiar.
  • Compact mode at 96px height; expanded editor/history mode at 280px.
  • Built as inline HTML/CSS/JS with a dedicated preload bridge.

• Chat History List UI

  • Replaced plain-text history textarea with styled per-entry cards.
  • Role badges (You / Familiar / System / Error), color coding, timestamps.
  • Persists to disk (familiaros-chat-history.json).

• Memory Viewer

  • New Settings section to search, edit, and delete FamiliarOS memories.

• MCP Tool Servers Panel

  • New top-level Integrations card to activate MCP tools for the built-in floating chat.
  • Works without Claude, Cursor, OpenCode, or Pi installed.
  • Tiered chip selector (Starter / Terminal & Systems / Advanced).

• FamiliarOS MCP Server Panel

  • New central Integrations card for the external-agent bridge.
  • Command mode, node path, and familiar routing in one place.
  • One-click server health test and MCP JSON copy.

• Text-to-Speech (Phase 2)

  • Settings > Text-to-Speech panel with provider, voice, model, speed, endpoint, and credential controls.
  • Providers: system OS voice, OpenAI TTS, ElevenLabs, Piper (local), and OpenAI-compatible (OpenRouter / LiteLLM / WaveSpeedAI / custom).
  • Credentials stored with safeStorage (plain fallback) and tied to the selected provider.
  • Shared TTS service (tts-service.ts) routes speech to the familiar / floating-chat renderer via familiaros:tts-speak, familiaros:tts-audio, and familiaros:tts-stop IPC.
  • Cloud audio validated for MP3/WAV; endpoints validated for HTTPS / localhost rules.
  • Skips speech during quiet hours.

New core modules

• apps/desktop/src/openapi-chat.ts — OpenAPI BYOK chat with responses + chat/completions support.
• apps/desktop/src/mcp-chat-client.ts — Internal stdio MCP client for floating chat tools.
• apps/desktop/src/familiaros-memory.ts — Local memory engine with JSON store and markdown mirror.
• apps/desktop/src/knowledge-store-core.ts — Electron-free file + memory knowledge repository with search and context building.
• apps/desktop/src/knowledge-store.ts — Electron wrapper that wires the core to app.getPath('userData').
• apps/desktop/src/tts-service.ts — Shared TTS dispatcher for familiar / floating chat with quiet-hours check.
• apps/desktop/src/tts-engine.ts — Provider-specific synthesis and voice fetching (OpenAI, ElevenLabs, Piper, OpenAI-compatible).
• apps/desktop/src/tts-catalog.ts — Provider catalog, defaults, and endpoint presets.
• apps/desktop/src/tts-credentials.ts — Secure credential storage for TTS providers.
• apps/desktop/src/mcp-toolkit-installer.ts — Toolkit install orchestration.
• apps/desktop/src/renderer/src/mcp-toolkit-catalog.ts — Catalog data for the toolkit UI.
• apps/desktop/prompt-window-preload.cjs — Preload for the floating chat renderer.

Familiar improvements

• Continuous familiar scale slider from 0.16x to 10x in Settings.
• Dynamic window sizing so larger scales are never clipped.
• Multi-screen positioning (clampToNearestWorkArea).
• Removed the on-familiar drag-to-resize handle in favor of the Settings slider.
• Virtual-familiar status (familiaros.virtual-familiar) moved from an on-familiar triple-click HUD overlay to the right-click familiar context menu, so it never blocks chat clicks. Single-clicking the familiar triggers the familiar() care action.

Settings additions

• OpenAPI chat: endpoint preset, model, credential, theme, base-instructions toggle.
• Moonshot/Kimi endpoint preset.
• Familiar character / system prompt field.
• Memory viewer with search/edit/delete.
• Knowledge Store tab: upload files, search stored knowledge, delete files, and add manual memory entries.

Bug fixes

• Memory capture/retrieval is now more robust in floating chat: capture is logged and errors are caught instead of breaking the send path.
• Speech bubble text no longer clips for long messages; bubble bodies scroll when content exceeds the bubble max-height.
• Fixed a packaged-build SyntaxError in the floating prompt window's inline script: newline literals in attachment separator strings are now correctly escaped so the script parses in the signed Windows installer.
• Verified the packaged prompt window with a Windows-hosted E2E smoke test over the reverse tunnel (CDP-driven: bridge present, input/send/attach elements render, typing enables send, no renderer exceptions).

Knowledge Store integration

• Files stored in Settings > Knowledge Store are indexed and can be searched from the same panel.
• Stored files are automatically considered when building OpenAPI chat instructions: relevant files are appended to the system prompt.
• Attachments in the floating prompt window can be stored directly into the Knowledge Store via the toolbar button.

Updated files

• README.md — added floating chat, memory, MCP Tool Servers, FamiliarOS MCP Server, and Curated MCP Toolkit bullets.
• FEATURES.md — comprehensive inventory of new capabilities.
• docs/pr-chat-memory-mcp-toolkit.md — implementation explainer.
• docs/phases/phase-14-familiar-scale-setting.md — expanded with new sizes and drag resize.
• apps/desktop/src/familiar-window.ts — removed scale handle; IPC handlers; dynamic window sizing.
• apps/desktop/familiar-preload.cjs — single/double-click detection; single-click emits familiar:clicked for virtual-familiar petting, double-click opens floating chat.
• apps/desktop/src/app-state-core.ts — expanded scale range.
• apps/desktop/src/renderer/src/main.tsx — Settings/Integrations UI additions.
• apps/desktop/src/windows.ts — memory IPC, toolkit IPC, preference handling.
• apps/desktop/tests/onboarding-state.test.ts — updated scale assertions.

New files

FEATURES.md
README_OUR_CHANGES.md
FEATURES_OUR_CHANGES.md
PULL_REQUEST.md
apps/desktop/prompt-window-preload.cjs
apps/desktop/src/mcp-chat-client.ts
apps/desktop/src/mcp-toolkit-installer.ts
apps/desktop/src/openapi-chat.ts
apps/desktop/src/familiaros-memory.ts
apps/desktop/src/prompt-window.ts
apps/desktop/src/renderer/src/mcp-toolkit-catalog.ts
apps/desktop/src/knowledge-store-core.ts
apps/desktop/src/knowledge-store.ts
apps/desktop/tests/knowledge-store.test.ts
apps/desktop/src/tts-service.ts
apps/desktop/src/tts-engine.ts
apps/desktop/src/tts-catalog.ts
apps/desktop/src/tts-credentials.ts
apps/desktop/tests/tts-engine.test.ts

How to test

pnpm install
pnpm --filter @familiaros/desktop build
pnpm --filter @familiaros/desktop test

All existing tests pass. Additional manual verification:

1. Launch the desktop app and double-click the familiar — floating chat should open.
2. Add an OpenAI-compatible API key in Settings > OpenAPI Chat.
3. Send a message; assistant reply should appear in the familiar bubble and in the history list.
4. Say "remember that my favorite color is blue" — memory should be captured.
5. Open Settings > Memory — the memory should be searchable, editable, and deletable.
6. Open Integrations > MCP Tool Servers — activate Filesystem + Fetch/Web, then chat "list my home directory files" — tools should run.
7. Open Integrations > FamiliarOS MCP Server — choose a command mode, click Test Server, and verify the server responds.
8. Open Integrations > Curated MCP Toolkit to browse copy-paste snippets and optional persistent bundles.
9. Open Settings > Text-to-Speech, choose System voice, click Speak — the familiar should speak the test phrase.
10. Add an OpenAI TTS key, select a voice, click Speak — audio should play through the familiar window.
11. Store a .txt file in Settings > Knowledge Store and ask the assistant about its content — the reply should reference the stored file.
12. Right-click the familiar → choose Virtual Familiar ▸ Feed/Play/Familiar/Nap; stats update and the context menu reflects the new values.
13. Change display — familiar should stay on the display it is currently near.

Backwards compatibility

• Existing MCP server behavior is unchanged.
• Existing agent integrations (Claude, OpenCode, Cursor, Pi) are unchanged.
• Existing plugin system is unchanged.
• TTS defaults to the system voice; no API key is required until a cloud provider is chosen.
• Preferences default safely: users without API keys see a clear "Add credential" state.

Notes for upstream

• Two documentation styles are provided:
  • Merged: README.md and FEATURES.md are updated in-place for easy adoption.
  • Isolated: README_OUR_CHANGES.md, FEATURES_OUR_CHANGES.md, and PULL_REQUEST.md describe only our additions so you can pick and choose.
• If you prefer smaller PRs, this can be split along these boundaries:
  1. Floating chat + OpenAPI BYOK
  2. Memory system + memory viewer
  3. Vanilla chat MCP tools
  4. Familiar sizing expansion + drag handle
  5. Multi-screen positioning