scriptoriumadmin/openpetswithchatandmcp
1
0
Fork 0
Code Issues Pull requests 6 Projects Releases Packages Wiki Activity Actions

feat(desktop): add Text-to-Speech Phase 2 and update docs/registry

Browse source 
- Add configurable TTS providers: system OS voice, OpenAI TTS, ElevenLabs,
  Piper (local), and OpenAI-compatible (OpenRouter/LiteLLM/WaveSpeedAI/custom).
- Add secure TTS credential storage, endpoint presets, voice/model/speed UI,
  and test/stop controls in Settings.
- Wire TTS to pet/floating-chat renderer via openpets:tts-speak/audio/stop IPC.
- Respect quiet hours; validate cloud endpoints and returned audio buffers.
- Add plugin voice:speak bridge and tts-engine unit tests.
- Update FEATURES.md, FEATURES_OUR_CHANGES.md, README.md, README_OUR_CHANGES.md,
  PULL_REQUEST.md, FEATURE_REGISTRY.md, and add docs/pull-requests/PR_TTS_PHASE_2.md.
This commit is contained in:
OpenPets Dev 2026-06-16 18:40:42 +00:00
parent 32f109f90b
commit bb758e9d0f
32 changed files with 1944 additions and 386 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

38
FEATURES.md
View file

@ -51,6 +51,23 @@ It combines:
- Safer speech rules for agent-driven bubble content
- Bubble behavior that avoids showing code, logs, URLs, paths, or secrets in normal integration speech
## Pet Text-to-Speech (Phase 2)
- Settings > **Text-to-Speech** panel controls voice output for the pet and floating chat.
- **System voice** uses the OS speech engine through the renderer (`window.speechSynthesis`) with optional voice-name matching and a 0.5×–2.0× speed multiplier.
- **Cloud TTS providers**: OpenAI TTS, ElevenLabs, and an **OpenAI-compatible** preset for OpenRouter, LiteLLM, WaveSpeedAI, or a custom endpoint.
- **Local TTS provider**: Piper (spawned as a local process) for fully offline speech.
- Provider credentials are stored with Electron `safeStorage` and fall back to plain local storage when encryption is unavailable (`apps/desktop/src/tts-credentials.ts`).
- Per-provider model, voice, speed, endpoint preset, and custom endpoint controls.
- Dynamic voice list fetching for ElevenLabs; static voice lists for OpenAI / OpenAI-compatible.
- In-app **Test voice** and **Stop** buttons to preview the configured voice.
- Assistant replies and other speech are spoken through the pet window renderer via a shared TTS service (`apps/desktop/src/tts-service.ts`).
- Respects quiet-hours: speech is skipped while quiet hours are active.
- Audio returned by cloud providers is validated (MP3/WAV magic bytes) and played through a renderer `<audio>` element.
- Endpoint validation enforces HTTPS for cloud providers, allows `http://localhost` for local proxies, and blocks private IP endpoints for cloud providers.
- Input capped at 2,000 characters; clear error messages when synthesis fails.
- Plugin SDK bridge: plugins granted the `voice:speak` permission can ask the pet to speak text through the same TTS pipeline.
## Virtual Pet Care
- Bundled `openpets.virtual-pet` plugin tracks hunger, energy, happiness, and affection
@ -184,6 +201,12 @@ It combines:
- `openpets_memory_search`
- `openpets_memory_store`
- `openpets_memory_forget`
- Central **OpenPets MCP Server** panel in Control Center → Integrations
- Command mode selector (published / bundled / local) in one place
- Node path override in the central panel
- Pet routing selector in the central panel
- One-click **Test Server** health check
- One-click **Copy MCP JSON** for external hosts
## Agent Integrations
@ -200,13 +223,16 @@ It combines:
- OpenCode plugin support
- Cursor MCP config preview
- Cursor rules preview
- Pet routing per supported integration
- Published package / bundled / local command modes for supported integrations
- Command path overrides for relevant integrations
- Pet routing centralized in the OpenPets MCP Server panel
- Published package / bundled / local command modes centralized in the OpenPets MCP Server panel
- Agent-specific command path overrides (Claude command, OpenCode command)
- Node command path centralized in the OpenPets MCP Server panel
## Vanilla Chat MCP Tools
## MCP Tool Servers (Built-in Chat)
- Multi-select MCP tool activation inside the floating chat window
- Top-level **MCP Tool Servers** panel in Control Center → Integrations
- Multi-select MCP tool activation for the built-in floating chat
- Works without Claude, Cursor, OpenCode, or Pi installed
- Tiered tool browser (Starter, Terminal & Systems, Advanced)
- Save & Activate / Deactivate All controls
- Tools run through an internal stdio MCP client in the main process
@ -214,7 +240,7 @@ It combines:
## Curated MCP Toolkit Surface
The control-center Integrations page now also includes a curated **MCP Toolkit** panel. This does not pretend to be a full unsafe one-click installer for the whole MCP ecosystem. Instead, it provides a practical, permission-aware reference surface with copyable snippets and docs links.
The control-center Integrations page now also includes a curated **Curated MCP Toolkit** panel. This does not pretend to be a full unsafe one-click installer for the whole MCP ecosystem. Instead, it provides a practical, permission-aware reference surface with copyable snippets and docs links.
### Starter Stack In The Toolkit

32
FEATURES_OUR_CHANGES.md
View file

@ -35,6 +35,19 @@ This document lists only the features, UI surfaces, and capabilities added by ou
- Character prompt persists in app state and applies to future assistant replies.
- Base instructions toggle to include/exclude the default OpenPets behavior instructions in every chat.
## Pet Text-to-Speech (Phase 2)
- Settings > **Text-to-Speech** panel for the pet and floating chat.
- **System voice** via the OS speech engine with voice matching and 0.5×–2.0× speed control.
- **Cloud providers**: OpenAI TTS, ElevenLabs, OpenAI-compatible (OpenRouter / LiteLLM / WaveSpeedAI / custom).
- **Local provider**: Piper for offline speech.
- Credentials stored with Electron `safeStorage` (plain fallback).
- Per-provider model, voice, speed, endpoint preset, and custom endpoint.
- Dynamic voice list fetching for ElevenLabs; static lists for OpenAI.
- Test voice + Stop buttons in Settings.
- Speech skips quiet hours, validates returned audio, and enforces HTTPS / localhost rules for endpoints.
- Plugin SDK `voice:speak` permission lets plugins speak through the same pipeline.
## Memory System
- Local-first durable OpenPets memory store.
@ -48,18 +61,28 @@ This document lists only the features, UI surfaces, and capabilities added by ou
- **Chat history memory**: cross-conversation message search that injects relevant past conversation excerpts into the system prompt.
- Separate databases for explicit memories (`openpets-memory.json`) and chat history (`openpets-chat-history.json`).
## Vanilla Chat MCP Tools
## MCP Tool Servers (Built-in Chat)
- Internal stdio MCP client in the desktop main process (`mcp-chat-client.ts`).
- Multi-select tool activation in the floating chat window.
- Top-level **MCP Tool Servers** panel in Control Center → Integrations.
- Multi-select tool activation for the built-in floating chat, independent of external agents.
- Tiered tool browser: Starter, Terminal & Systems, Advanced.
- Save & Activate / Deactivate All controls.
- ReAct-style tool-calling loop (max 5 iterations) with responses/chat-completions integration.
- Supported tools: filesystem, terminal/shell, memory, fetch-web, sequential-thinking, playwright, git, github, docker, sqlite.
## OpenPets MCP Server (External-Agent Bridge)
- Central **OpenPets MCP Server** panel in Control Center → Integrations.
- Command mode selector (published / bundled / local) in one place.
- Node path override in the central panel.
- Pet routing selector in the central panel.
- One-click **Test Server** health check.
- One-click **Copy MCP JSON** for external hosts.
## Curated MCP Toolkit Integration
- Curated MCP Toolkit panel inside Control Center > Integrations.
- Curated MCP Toolkit panel inside Control Center → Integrations.
- Multi-select chips by tier (Starter, Terminal & Systems, Advanced).
- Command Source labels clarify where tools come from:
- Stable release (npm)
@ -106,7 +129,8 @@ This document lists only the features, UI surfaces, and capabilities added by ou
## Settings & Control Center
- Memory viewer section in Settings.
- Vanilla Chat tool activation section in Integrations.
- MCP Tool Servers panel in Integrations for built-in chat tool activation.
- OpenPets MCP Server panel in Integrations for external-agent bridge configuration.
- OpenAPI chat settings: endpoint, model, credential, theme, base instructions toggle.
- Moonshot/Kimi API endpoint preset.

37
FEATURE_REGISTRY.md
View file

@ -3,7 +3,7 @@
This is the canonical feature registry for the VectorShell working fork of OpenPets. It maps every user-facing capability to the files, branches/PRs, and commits that implement it. It is kept in addition to `FEATURES.md` and `FEATURES_OUR_CHANGES.md` so that feature ownership, status, and history can be traced quickly.
- **Current branch:** `v3.1.0-integrated`
- **Current HEAD:** `16bfab1`
- **Current HEAD:** `8369418`
- **Version:** `3.1.0`
- **Status key:** `shipped` = in the signed Windows installer; `merged` = in `v3.1.0-integrated`; `in-review` = has a PR branch; `experimental` = behind a flag or not yet packaged.
@ -83,16 +83,25 @@ This is the canonical feature registry for the VectorShell working fork of OpenP
| Memory tools (list/search/store/forget) | shipped | upstream | — | `packages/mcp/src/tools.ts` | Baseline. |
| Lease routing + default-pet fallback | shipped | upstream | — | `packages/mcp/src/server.ts` | Baseline. |
### 6b. Vanilla chat MCP tools
### 6b. MCP Tool Servers (built-in chat)
| Feature | Status | PR / Branch | Key commits | Key files | Notes |
|---------|--------|-------------|-------------|-----------|-------|
| Internal stdio MCP client | shipped | main PR | `08eff41` | `apps/desktop/src/mcp-chat-client.ts` | Spawns tool servers inside floating chat. |
| Tool activation UI | shipped | main PR | `08eff41` | `apps/desktop/src/renderer/src/main.tsx` | Multi-select chips by tier. |
| Top-level "MCP Tool Servers" panel | shipped | main PR | current | `apps/desktop/src/renderer/src/main.tsx` | Central place to activate tool servers for built-in chat, independent of external agents. |
| Tool activation UI | shipped | main PR | `08eff41` / current | `apps/desktop/src/renderer/src/main.tsx` | Multi-select chips by tier. |
| Supported tool servers | shipped | main PR | `08eff41` | `apps/desktop/src/mcp-chat-client.ts` | filesystem, terminal, memory, fetch-web, sequential-thinking, playwright, git, github, docker, sqlite. |
| ReAct-style tool loop | shipped | main PR | `08eff41` | `apps/desktop/src/mcp-chat-client.ts`, `apps/desktop/src/openapi-chat.ts` | Max 5 iterations. |
### 6c. Curated MCP Toolkit
### 6c. OpenPets MCP Server (external-agent bridge)
| Feature | Status | PR / Branch | Key commits | Key files | Notes |
|---------|--------|-------------|-------------|-----------|-------|
| Central "OpenPets MCP Server" panel | shipped | main PR | current | `apps/desktop/src/renderer/src/main.tsx`, `apps/desktop/src/agent-setup.ts` | Configures command mode, node path, pet routing, test server, and MCP JSON for external hosts. |
| Server health test | shipped | main PR | current | `apps/desktop/src/agent-setup.ts`, `apps/desktop/src/windows.ts` | `--version` smoke test via IPC. |
| MCP JSON copy | shipped | main PR | current | `apps/desktop/src/renderer/src/main.tsx` | One-click copy of the `mcpServers.openpets` entry. |
### 6d. Curated MCP Toolkit
| Feature | Status | PR / Branch | Key commits | Key files | Notes |
|---------|--------|-------------|-------------|-----------|-------|
@ -118,7 +127,22 @@ This is the canonical feature registry for the VectorShell working fork of OpenP
| Official plugin suite | shipped | upstream | — | `plugins/official/` | Baseline: focus-buddy, reminders, launch-buddy, water-reminder, mood-check-in, day-routine, magic-8-ball, fortune-cookie, virtual-pet. |
| Virtual-pet plugin bundling | shipped | `pr/virtual-pet-context-menu` | `98b7875` | `plugins/official/openpets.virtual-pet/index.js` | Re-enabled after temporary disable. |
## 9. Packaging / release
## 9. Text-to-Speech (Phase 2)
| Feature | Status | PR / Branch | Key commits | Key files | Notes |
|---------|--------|-------------|-------------|-----------|-------|
| TTS settings panel | merged | TTS Phase 2 / `v3.1.0-integrated` | current | `apps/desktop/src/renderer/src/main.tsx` | Provider, voice, speed, model, endpoint preset, credential, and test/stop UI. |
| System voice playback | merged | TTS Phase 2 / `v3.1.0-integrated` | current | `apps/desktop/pet-preload.cjs`, `apps/desktop/prompt-window-preload.cjs` | OS speech engine via Web Speech API; rate 0.5×–2.0× and voice-name matching. |
| Cloud TTS providers | merged | TTS Phase 2 / `v3.1.0-integrated` | current | `apps/desktop/src/tts-engine.ts` | OpenAI TTS, ElevenLabs, and OpenAI-compatible endpoints. |
| Piper local TTS | merged | TTS Phase 2 / `v3.1.0-integrated` | current | `apps/desktop/src/tts-engine.ts` | Spawns local `piper` binary; WAV magic-byte validation. |
| Credential storage | merged | TTS Phase 2 / `v3.1.0-integrated` | current | `apps/desktop/src/tts-credentials.ts` | Per-provider API key stored with `safeStorage`; plain fallback. |
| Endpoint presets | merged | TTS Phase 2 / `v3.1.0-integrated` | current | `apps/desktop/src/tts-catalog.ts` | OpenRouter, LiteLLM, WaveSpeedAI, Custom. |
| TTS service & quiet hours | merged | TTS Phase 2 / `v3.1.0-integrated` | current | `apps/desktop/src/tts-service.ts` | Shared dispatcher; skips speech during quiet hours. |
| TTS IPC handlers | merged | TTS Phase 2 / `v3.1.0-integrated` | current | `apps/desktop/src/windows.ts` | `openpets:get-tts-settings`, `openpets:get-tts-voices`, `openpets:save-tts-credential`, `openpets:test-tts`, `openpets:tts-stop`. |
| TTS engine tests | merged | TTS Phase 2 / `v3.1.0-integrated` | current | `apps/desktop/tests/tts-engine.test.ts` | Provider catalog, defaults, and preset validation. |
| Plugin voice bridge | merged | TTS Phase 2 / `v3.1.0-integrated` | current | `apps/desktop/src/plugin-voice.ts` | `voice:speak` permission lets plugins speak through the same pipeline. |
## 10. Packaging / release
| Feature | Status | PR / Branch | Key commits | Key files | Notes |
|---------|--------|-------------|-------------|-----------|-------|
@ -159,6 +183,7 @@ This is the canonical feature registry for the VectorShell working fork of OpenP
| `docs/pull-requests/PR_SCALE_SLIDER_DRAGGABLE.md` | Scale slider draggable thumb PR. |
| `docs/pull-requests/PR_PACKAGING_AND_DOCS.md` | Packaging/docs PR. |
| `docs/pull-requests/PROMPT_WINDOW_PACKAGED_BUILD_FIX_2026-06-15.md` | Prompt-window packaged-build fix root cause. |
| `docs/pull-requests/PR_TTS_PHASE_2.md` | Text-to-Speech Phase 2 PR description. |
| `docs/pr-chat-memory-mcp-toolkit.md` | Floating chat + memory + MCP toolkit explainer. |
---
@ -166,7 +191,7 @@ This is the canonical feature registry for the VectorShell working fork of OpenP
## Branches on this repo
- `main` — upstream-aligned baseline.
- `v3.1.0-integrated` — integration branch with all v3.1.0 work (HEAD `16bfab1`).
- `v3.1.0-integrated` — integration branch with all v3.1.0 work plus TTS Phase 2 (HEAD `8369418`).
- `feat/scale-preset-dropdown`, `fix/pet-window-stability`, `fix/prompt-window-hardening`, `fix/prompt-window-signed-build`, `fix/scale-handle-and-keyboard`, `refactor/virtual-pet-context-menu` — topic branches from the six original PRs.
- `op1/pet-drag-threshold` … `op6/chat-file-attachments` — user-reported tweak branches.
- `pr/bubble-clip-fix`, `pr/memory-hardening`, `pr/packaging-and-docs`, `pr/scale-slider-draggable`, `pr/single-click-petting`, `pr/virtual-pet-context-menu` — PR branch pointers.

46
PULL_REQUEST.md
View file

@ -1,6 +1,6 @@
# Pull Request: Floating Chat, Memory, Knowledge Store, MCP Toolkit & Pet Sizing
# Pull Request: Floating Chat, Memory, Knowledge Store, MCP Servers, Curated MCP Toolkit, TTS Phase 2 & Pet Sizing
**Title:** feat(desktop): add floating chat, local memory, Knowledge Store, vanilla MCP tools, expanded pet sizing, and multi-screen support
**Title:** feat(desktop): add floating chat, local memory, Knowledge Store, MCP Tool Servers, OpenPets MCP Server panel, Curated MCP Toolkit, Text-to-Speech Phase 2, expanded pet sizing, and multi-screen support
**Author:** VectorShell working fork
**Base:** upstream `main` at `cba81b7`
@ -36,10 +36,24 @@ This PR adds a complete in-app chat layer, a durable local memory system, intera
- **Memory Viewer**
- New Settings section to search, edit, and delete OpenPets memories.
- **Vanilla Chat MCP Tool Activation**
- New Integrations UI to activate MCP tools for use inside floating chat.
- **MCP Tool Servers Panel**
- New top-level Integrations card to activate MCP tools for the built-in floating chat.
- Works without Claude, Cursor, OpenCode, or Pi installed.
- Tiered chip selector (Starter / Terminal & Systems / Advanced).
- **OpenPets MCP Server Panel**
- New central Integrations card for the external-agent bridge.
- Command mode, node path, and pet routing in one place.
- One-click server health test and MCP JSON copy.
- **Text-to-Speech (Phase 2)**
- Settings > **Text-to-Speech** panel with provider, voice, model, speed, endpoint, and credential controls.
- Providers: system OS voice, OpenAI TTS, ElevenLabs, Piper (local), and OpenAI-compatible (OpenRouter / LiteLLM / WaveSpeedAI / custom).
- Credentials stored with `safeStorage` (plain fallback) and tied to the selected provider.
- Shared TTS service (`tts-service.ts`) routes speech to the pet / floating-chat renderer via `openpets:tts-speak`, `openpets:tts-audio`, and `openpets:tts-stop` IPC.
- Cloud audio validated for MP3/WAV; endpoints validated for HTTPS / localhost rules.
- Skips speech during quiet hours.
### New core modules
- `apps/desktop/src/openapi-chat.ts` — OpenAPI BYOK chat with responses + chat/completions support.
@ -47,6 +61,10 @@ This PR adds a complete in-app chat layer, a durable local memory system, intera
- `apps/desktop/src/openpets-memory.ts` — Local memory engine with JSON store and markdown mirror.
- `apps/desktop/src/knowledge-store-core.ts` — Electron-free file + memory knowledge repository with search and context building.
- `apps/desktop/src/knowledge-store.ts` — Electron wrapper that wires the core to `app.getPath('userData')`.
- `apps/desktop/src/tts-service.ts` — Shared TTS dispatcher for pet / floating chat with quiet-hours check.
- `apps/desktop/src/tts-engine.ts` — Provider-specific synthesis and voice fetching (OpenAI, ElevenLabs, Piper, OpenAI-compatible).
- `apps/desktop/src/tts-catalog.ts` — Provider catalog, defaults, and endpoint presets.
- `apps/desktop/src/tts-credentials.ts` — Secure credential storage for TTS providers.
- `apps/desktop/src/mcp-toolkit-installer.ts` — Toolkit install orchestration.
- `apps/desktop/src/renderer/src/mcp-toolkit-catalog.ts` — Catalog data for the toolkit UI.
- `apps/desktop/prompt-window-preload.cjs` — Preload for the floating chat renderer.
@ -82,7 +100,7 @@ This PR adds a complete in-app chat layer, a durable local memory system, intera
### Updated files
- `README.md` — added floating chat, memory, and vanilla chat tool bullets.
- `README.md` — added floating chat, memory, MCP Tool Servers, OpenPets MCP Server, and Curated MCP Toolkit bullets.
- `FEATURES.md` — comprehensive inventory of new capabilities.
- `docs/pr-chat-memory-mcp-toolkit.md` — implementation explainer.
- `docs/phases/phase-14-pet-scale-setting.md` — expanded with new sizes and drag resize.
@ -110,6 +128,11 @@ apps/desktop/src/renderer/src/mcp-toolkit-catalog.ts
apps/desktop/src/knowledge-store-core.ts
apps/desktop/src/knowledge-store.ts
apps/desktop/tests/knowledge-store.test.ts
apps/desktop/src/tts-service.ts
apps/desktop/src/tts-engine.ts
apps/desktop/src/tts-catalog.ts
apps/desktop/src/tts-credentials.ts
apps/desktop/tests/tts-engine.test.ts
```
## How to test
@ -127,16 +150,21 @@ All existing tests pass. Additional manual verification:
3. Send a message; assistant reply should appear in the pet bubble and in the history list.
4. Say "remember that my favorite color is blue" — memory should be captured.
5. Open Settings > Memory — the memory should be searchable, editable, and deletable.
6. Open Integrations > MCP Toolkit — activate Filesystem + Fetch/Web, then chat "list my home directory files" — tools should run.
7. Store a `.txt` file in Settings > Knowledge Store and ask the assistant about its content — the reply should reference the stored file.
8. Right-click the pet → choose Virtual Pet ▸ Feed/Play/Pet/Nap; stats update and the context menu reflects the new values.
8. Change display — pet should stay on the display it is currently near.
6. Open Integrations > MCP Tool Servers — activate Filesystem + Fetch/Web, then chat "list my home directory files" — tools should run.
7. Open Integrations > OpenPets MCP Server — choose a command mode, click Test Server, and verify the server responds.
8. Open Integrations > Curated MCP Toolkit to browse copy-paste snippets and optional persistent bundles.
9. Open Settings > Text-to-Speech, choose System voice, click **Speak** — the pet should speak the test phrase.
10. Add an OpenAI TTS key, select a voice, click **Speak** — audio should play through the pet window.
11. Store a `.txt` file in Settings > Knowledge Store and ask the assistant about its content — the reply should reference the stored file.
12. Right-click the pet → choose Virtual Pet ▸ Feed/Play/Pet/Nap; stats update and the context menu reflects the new values.
13. Change display — pet should stay on the display it is currently near.
## Backwards compatibility
- Existing MCP server behavior is unchanged.
- Existing agent integrations (Claude, OpenCode, Cursor, Pi) are unchanged.
- Existing plugin system is unchanged.
- TTS defaults to the system voice; no API key is required until a cloud provider is chosen.
- Preferences default safely: users without API keys see a clear "Add credential" state.
## Notes for upstream

18
README.md
View file

@ -11,7 +11,7 @@
</p>
<p align="center">
<sub>This branch also includes a <strong>floating chat surface</strong>, <strong>local memory</strong>, <strong>Knowledge Store</strong>, <strong>vanilla MCP tools</strong>, and <strong>expanded pet sizing</strong>. See <code>FEATURES.md</code>, <code>FEATURE_REGISTRY.md</code>, and <code>PULL_REQUEST.md</code> for details.</sub>
<sub>This branch also includes a <strong>floating chat surface</strong>, <strong>local memory</strong>, <strong>Knowledge Store</strong>, <strong>MCP Tool Servers</strong>, an <strong>OpenPets MCP Server</strong> panel, a <strong>Curated MCP Toolkit</strong>, <strong>Text-to-Speech (Phase 2)</strong>, and <strong>expanded pet sizing</strong>. See <code>FEATURES.md</code>, <code>FEATURE_REGISTRY.md</code>, and <code>PULL_REQUEST.md</code> for details.</sub>
</p>
<p align="center">
@ -77,10 +77,11 @@ OpenPets is a tray-first desktop companion app. A pet appears on your desktop, k
- **Bundled abilities** - first-party plugins can add ambient check-ins, break nudges, playful pet actions, focus timers, safe little walks, and optional developer notifications.
- **Developer integrations** - advanced setup for Claude Code, OpenCode, Cursor, Pi, and MCP-capable tools when you want coding activity to drive the pet.
- **MCP ready** - any MCP-capable agent can send short safe speech bubbles and reactions through the OpenPets MCP server.
- **Floating chat** - double-click the pet to open an always-on-top chat window with BYOK OpenAPI-compatible providers, conversation history, and optional MCP tool activation.
- **Floating chat** - double-click the pet to open an always-on-top chat window with BYOK OpenAPI-compatible providers, conversation history, and optional MCP Tool Server activation.
- **Virtual-pet status** - right-click the pet and open the Virtual Pet submenu to see hunger, energy, happiness, bond, mood, and level. No on-pet HUD overlay, so chat clicks stay unobstructed.
- **Single-click petting** - click the pet once to pet it; the virtual-pet bond and happiness increase.
- **Local memory** - the pet remembers facts, preferences, and notes across sessions.
- **Text-to-Speech (Phase 2)** - configurable system, OpenAI, ElevenLabs, Piper, or OpenAI-compatible TTS for the pet and floating chat, with voice/model/speed controls and a test preview.
- **Pet-pack friendly** - loads installed animated pet packs and can route a selected agent/project to its own pet window.
- **Privacy-conscious by design** - automatic hook speech is static and local; prompts, code, logs, command output, URLs, paths, and secrets are not shown in bubbles.
@ -229,15 +230,22 @@ Available MCP tools:
`openpets_say` messages must be short, single-line, and must not look like code, logs, secrets, URLs, or file paths.
### MCP Tool Servers
The desktop app includes a dedicated **MCP Tool Servers** panel under **Integrations** for the built-in floating chat. Activate tool servers such as filesystem, terminal, memory, fetch-web, sequential-thinking, playwright, git, github, docker, and sqlite so the pet can use them directly. This works independently of whether Claude, Cursor, OpenCode, or Pi are installed.
### OpenPets MCP Server
A central **OpenPets MCP Server** panel under **Integrations** configures the external-agent bridge. It collects command mode (published / bundled / local), node path, and pet routing in one place, and provides a **Test Server** check and a **Copy MCP JSON** button for pasting into Claude Code, Cursor, Codex CLI, or any other MCP host.
### Curated MCP toolkit
The desktop app also includes a manual-but-curated **MCP Toolkit** panel under **Integrations**. It is there to help you build a practical broader stack without stuffing OpenPets full of unsafe one-click third-party installs.
The desktop app also includes a manual-but-curated **Curated MCP Toolkit** panel under **Integrations**. It is there to help you build a practical broader stack without stuffing OpenPets full of unsafe one-click third-party installs.
The toolkit now explicitly offers two paths:
The toolkit offers two paths:
- `Manual Setup` - per-tool guidance, docs links, and copyable snippets
- `Persistent Full Access` - `Install Now` for the supported persistent baseline in hosts such as Claude Code or Codex CLI, plus a matching copyable bundle
- `Vanilla Chat Tools` - activate MCP tools directly inside the floating chat window for autonomous filesystem, terminal, web, and browser tasks
The current supported `Install Now` baseline covers:

31
README_OUR_CHANGES.md
View file

@ -33,14 +33,24 @@ Store files and add manual knowledge entries that the assistant can reference du
- Relevant files are injected into the assistant's system instructions automatically.
- Combined search across stored files and OpenPets memories.
## New: Vanilla Chat MCP Tools
## New: MCP Tool Servers (Built-in Chat)
Activate MCP tools directly inside the floating chat window for autonomous task execution.
Activate MCP tool servers for the built-in floating chat, independently of external agents.
- Filesystem, Terminal, Memory, Fetch/Web, Sequential Thinking, Playwright, Git, GitHub, Docker, SQLite.
- Tiered browser in Settings/Integrations: Starter, Terminal & Systems, Advanced.
- Top-level **MCP Tool Servers** panel in Control Center → Integrations.
- Tiered browser: Starter, Terminal & Systems, Advanced.
- Tools run through an internal stdio MCP client in the main process.
## New: OpenPets MCP Server Panel
A central panel in Control Center → Integrations configures the external-agent bridge.
- Command mode selector (published / bundled / local) in one place.
- Node path override and pet routing in one place.
- One-click **Test Server** health check.
- One-click **Copy MCP JSON** for Claude Code, Cursor, Codex CLI, or generic MCP hosts.
## New: Pet Scale
- Continuous scale slider from 0.16x to 10x in Settings.
@ -57,6 +67,16 @@ Activate MCP tools directly inside the floating chat window for autonomous task
- Give your pet a custom system-prompt style character in Settings.
- Toggle whether the default OpenPets behavior instructions are included in every chat.
## New: Text-to-Speech (Phase 2)
The pet can now speak aloud using a configurable TTS provider.
- Settings > **Text-to-Speech** chooses between system OS voice, OpenAI TTS, ElevenLabs, Piper (local), or an OpenAI-compatible endpoint.
- Per-provider voice, model, speed, endpoint preset, and API-key storage.
- Test and Stop buttons preview the voice in Settings.
- Speech is skipped during quiet hours and uses the same safety boundaries as bubbles.
- Plugins with the `voice:speak` permission can ask the pet to speak.
## New: Multi-Screen Awareness
The pet now stays on the display it is currently near, rather than always snapping to the primary display.
@ -69,6 +89,9 @@ No additional setup is required beyond the normal OpenPets desktop app. The new
- Double-click the pet → floating chat.
- Right-click the pet → Virtual Pet status and care commands.
- Settings → OpenAPI Chat for model/credential setup.
- Settings → Text-to-Speech for voice provider and test preview.
- Settings → Memory for memory management.
- Settings → Knowledge Store for file-based knowledge.
- Integrations → MCP Toolkit for tool activation.
- Integrations → MCP Tool Servers for built-in chat tool activation.
- Integrations → OpenPets MCP Server for external-agent bridge configuration.
- Integrations → Curated MCP Toolkit for optional persistent bundles and copy-paste snippets.

8
apps/desktop/control-center-preload.cjs
View file

@ -55,6 +55,14 @@ const api = {
getIntegrationsState: (selectedPetId, commandMode) => ipcRenderer.invoke("openpets:agent-setup-snapshot", selectedPetId, commandMode),
runIntegrationAction: (action, selectedPetId, commandMode) => ipcRenderer.invoke("openpets:agent-setup-action", action, selectedPetId, commandMode),
updateIntegrationCommandPaths: (patch) => ipcRenderer.invoke("openpets:agent-setup-command-paths", patch),
getOpenPetsMcpServerPreview: (selectedPetId, commandMode) => ipcRenderer.invoke("openpets:openpets-mcp-server-preview", selectedPetId, commandMode),
testOpenPetsMcpServer: (selectedPetId, commandMode) => ipcRenderer.invoke("openpets:test-openpets-mcp-server", selectedPetId, commandMode),
getTtsSettings: () => ipcRenderer.invoke("openpets:get-tts-settings"),
getTtsVoices: (provider) => ipcRenderer.invoke("openpets:get-tts-voices", provider),
saveTtsCredential: (provider, credential) => ipcRenderer.invoke("openpets:save-tts-credential", provider, credential),
clearTtsCredential: (provider) => ipcRenderer.invoke("openpets:clear-tts-credential", provider),
testTtsSpeak: (text) => ipcRenderer.invoke("openpets:test-tts", text),
stopTts: () => ipcRenderer.invoke("openpets:tts-stop"),
copyText: (text) => ipcRenderer.invoke("openpets:copy-text", text),
openExternalUrl: (url) => ipcRenderer.invoke("openpets:open-external-url", url),
installMcpToolkit: (target) => ipcRenderer.invoke("openpets:install-mcp-toolkit", target),

34
apps/desktop/pet-preload.cjs
View file

@ -395,8 +395,11 @@ ipcRenderer.on("openpets:stop-audio", () => {
// --- Plugin TTS ---------------------------------------------------------------
let currentTtsAudio = null;
ipcRenderer.on("openpets:tts-speak", (_event, payload) => {
try {
stopCurrentTtsAudio();
if (!payload || typeof payload.text !== "string" || !window.speechSynthesis) return;
const utterance = new SpeechSynthesisUtterance(payload.text.slice(0, 500));
if (typeof payload.rate === "number" && payload.rate >= 0.5 && payload.rate <= 2) utterance.rate = payload.rate;
@ -408,10 +411,37 @@ ipcRenderer.on("openpets:tts-speak", (_event, payload) => {
} catch { /* tts is best-effort */ }
});
ipcRenderer.on("openpets:tts-stop", () => {
try { window.speechSynthesis && window.speechSynthesis.cancel(); } catch { /* noop */ }
ipcRenderer.on("openpets:tts-audio", (_event, payload) => {
try {
stopCurrentTtsAudio();
if (!payload || !payload.audio || !payload.mimeType) return;
const buffer = payload.audio instanceof Uint8Array ? payload.audio : new Uint8Array(payload.audio);
const blob = new Blob([buffer], { type: payload.mimeType });
const url = URL.createObjectURL(blob);
currentTtsAudio = new Audio(url);
currentTtsAudio.addEventListener("ended", () => {
URL.revokeObjectURL(url);
currentTtsAudio = null;
});
currentTtsAudio.play().catch(() => { /* audio is best-effort */ });
} catch { /* tts is best-effort */ }
});
ipcRenderer.on("openpets:tts-stop", () => {
try {
stopCurrentTtsAudio();
if (window.speechSynthesis) window.speechSynthesis.cancel();
} catch { /* noop */ }
});
function stopCurrentTtsAudio() {
if (currentTtsAudio) {
try { currentTtsAudio.pause(); } catch { /* noop */ }
try { currentTtsAudio.src = ""; } catch { /* noop */ }
currentTtsAudio = null;
}
}
const installMouseInterop = () => {
lastInteractiveHit = null;
dragging = false;

45
apps/desktop/prompt-window-preload.cjs
View file

@ -15,3 +15,48 @@ const api = {
};
contextBridge.exposeInMainWorld("openPetsPromptWindow", api);
// --- TTS playback -------------------------------------------------------------
let currentTtsAudio = null;
ipcRenderer.on("openpets:tts-speak", (_event, payload) => {
try {
stopCurrentTtsAudio();
if (!payload || typeof payload.text !== "string" || !window.speechSynthesis) return;
const utterance = new SpeechSynthesisUtterance(payload.text.slice(0, 500));
if (typeof payload.rate === "number" && payload.rate >= 0.5 && payload.rate <= 2) utterance.rate = payload.rate;
window.speechSynthesis.speak(utterance);
} catch { /* tts is best-effort */ }
});
ipcRenderer.on("openpets:tts-audio", (_event, payload) => {
try {
stopCurrentTtsAudio();
if (!payload || !payload.audio || !payload.mimeType) return;
const buffer = payload.audio instanceof Uint8Array ? payload.audio : new Uint8Array(payload.audio);
const blob = new Blob([buffer], { type: payload.mimeType });
const url = URL.createObjectURL(blob);
currentTtsAudio = new Audio(url);
currentTtsAudio.addEventListener("ended", () => {
URL.revokeObjectURL(url);
currentTtsAudio = null;
});
currentTtsAudio.play().catch(() => { /* audio is best-effort */ });
} catch { /* tts is best-effort */ }
});
ipcRenderer.on("openpets:tts-stop", () => {
try {
stopCurrentTtsAudio();
if (window.speechSynthesis) window.speechSynthesis.cancel();
} catch { /* noop */ }
});
function stopCurrentTtsAudio() {
if (currentTtsAudio) {
try { currentTtsAudio.pause(); } catch { /* noop */ }
try { currentTtsAudio.src = ""; } catch { /* noop */ }
currentTtsAudio = null;
}
}

1
apps/desktop/scripts/run-tests.mjs
View file

@ -19,6 +19,7 @@ const behaviorTests = [
".test-dist/tests/update-version.test.js",
".test-dist/tests/reaction-animation-mapping.test.js",
".test-dist/tests/zip-safety.test.js",
".test-dist/tests/tts-engine.test.js",
".test-dist/tests/codex-pets.test.js",
".test-dist/tests/claude-memory.test.js",
".test-dist/tests/prompt-memory-extraction.test.js",

52
apps/desktop/src/agent-setup.ts
View file

@ -4,7 +4,7 @@ import { dirname, isAbsolute, join } from "node:path";
import { createRequire } from "node:module";
import { app } from "electron";
import { buildClaudeMcpGetCommand, buildClaudeMcpPreview, classifyClaudeMcpStatus, createOpenPetsHookSettingsPreview, doctorClaudeHooks, installClaudeHooks, mapAsarPathToUnpacked, uninstallClaudeHooks, type ClaudeCommandSpec, type ClaudeHookDoctorResult, type ClaudeMcpPreview, type OpenPetsCommandMode, type ParsedClaudeMcpEntry } from "@open-pets/claude";
import { buildClaudeMcpGetCommand, buildClaudeMcpPreview, buildOpenPetsMcpServerCommand, classifyClaudeMcpStatus, createOpenPetsHookSettingsPreview, doctorClaudeHooks, formatCommandForDisplay, installClaudeHooks, mapAsarPathToUnpacked, uninstallClaudeHooks, type ClaudeCommandSpec, type ClaudeHookDoctorResult, type ClaudeMcpPreview, type OpenPetsCommandMode, type ParsedClaudeMcpEntry } from "@open-pets/claude";
import { buildCursorRulesPreview, classifyCursorMcpStatus, executeCursorMcpWrite, getCursorGlobalMcpPath, planCursorMcpInstall, planCursorMcpRemove, planCursorMcpReplace, readCursorMcpConfig, type CursorMcpStatusResult } from "@open-pets/cursor";
import { buildOpenPetsOnlyPreview, type RedactedPreview } from "@open-pets/cursor";
import { doctorOpenCodeGlobalSetup, getGlobalOpenCodeConfigDir, parseOpenCodeConfig, prepareOpenCodeGlobalRemove, prepareOpenCodeGlobalSetup, writePreparedOpenCodeGlobalRemove, writePreparedOpenCodeGlobalSetup } from "@open-pets/opencode";
@ -607,6 +607,56 @@ function getMcpPackageVersion(): string {
return getWorkspacePackageVersion("@open-pets/mcp");
}
export interface OpenPetsMcpServerPreview {
readonly commandMode: OpenPetsCommandMode;
readonly command: string;
readonly args: readonly string[];
readonly displayCommand: string;
readonly mcpJson: {
readonly mcpServers: {
readonly openpets: {
readonly type: "stdio";
readonly command: string;
readonly args: readonly string[];
};
};
};
}
export interface OpenPetsMcpServerHealth {
readonly ok: boolean;
readonly output: string;
readonly error?: string;
}
export function buildOpenPetsMcpServerPreview(selectedPetId: string | undefined, commandMode: OpenPetsCommandMode): OpenPetsMcpServerPreview {
const nodeCommand = getPreferredNodeCommand();
const { command, args } = buildOpenPetsMcpServerCommand(selectedPetId, commandMode, nodeCommand);
const displayCommand = formatCommandForDisplay({ command, args });
return {
commandMode,
command,
args,
displayCommand,
mcpJson: {
mcpServers: {
openpets: { type: "stdio", command, args },
},
},
};
}
export async function testOpenPetsMcpServer(selectedPetId: string | undefined, commandMode: OpenPetsCommandMode): Promise<OpenPetsMcpServerHealth> {
const nodeCommand = getPreferredNodeCommand();
const { command, args } = buildOpenPetsMcpServerCommand(selectedPetId, commandMode, nodeCommand);
const result = await runCommand({ command, args: [...args, "--version"] });
if (result.ok) {
return { ok: true, output: result.stdout.trim() || "MCP server responded." };
}
const errorMessage = result.error || result.stderr.trim() || `Command exited with code ${result.exitCode ?? "unknown"}.`;
return { ok: false, output: "", error: errorMessage };
}
function summarizeMemoryMessages(...messages: readonly string[]): string {
const memoryMessages = messages.flatMap((message) => message.match(/Claude (?:OpenPets )?instructions[^.]*\./g) ?? []);
return memoryMessages.length > 0 ? ` ${memoryMessages.join(" ")}` : "";

18
apps/desktop/src/app-state.ts
View file

@ -51,6 +51,12 @@ export interface OpenPetsStateV1 {
readonly opencodeCommandPath?: string;
readonly vanillaChatMcpTools?: readonly string[];
readonly openApiChatBaseInstructionsEnabled?: boolean;
readonly ttsProvider?: "system" | "openai" | "elevenlabs" | "piper" | "openai-compatible";
readonly ttsVoice?: string;
readonly ttsSpeed?: number;
readonly ttsModel?: string;
readonly ttsEndpointPreset?: "openrouter" | "litellm" | "wavespeedai" | "custom";
readonly ttsEndpoint?: string;
};
readonly pets: {
readonly installed: readonly InstalledPetState[];
@ -407,6 +413,12 @@ function normalizePreferences(value: Partial<OpenPetsStateV1["preferences"]>): O
opencodeCommandPath: normalizeCommandPath(value.opencodeCommandPath),
vanillaChatMcpTools: normalizeVanillaChatMcpTools(value.vanillaChatMcpTools),
openApiChatBaseInstructionsEnabled: typeof value.openApiChatBaseInstructionsEnabled === "boolean" ? value.openApiChatBaseInstructionsEnabled : true,
ttsProvider: ["system", "openai", "elevenlabs", "piper", "openai-compatible"].includes(value.ttsProvider as string) ? (value.ttsProvider as OpenPetsStateV1["preferences"]["ttsProvider"]) : "system",
ttsVoice: typeof value.ttsVoice === "string" ? value.ttsVoice : undefined,
ttsSpeed: typeof value.ttsSpeed === "number" && value.ttsSpeed >= 0.5 && value.ttsSpeed <= 2 ? value.ttsSpeed : 1,
ttsModel: typeof value.ttsModel === "string" ? value.ttsModel : undefined,
ttsEndpointPreset: ["openrouter", "litellm", "wavespeedai", "custom"].includes(value.ttsEndpointPreset as string) ? (value.ttsEndpointPreset as OpenPetsStateV1["preferences"]["ttsEndpointPreset"]) : "openrouter",
ttsEndpoint: typeof value.ttsEndpoint === "string" ? value.ttsEndpoint : undefined,
};
}
@ -543,6 +555,12 @@ function createDefaultState(): OpenPetsStateV1 {
opencodeCommandPath: undefined,
vanillaChatMcpTools: undefined,
openApiChatBaseInstructionsEnabled: true,
ttsProvider: "system",
ttsVoice: undefined,
ttsSpeed: 1,
ttsModel: undefined,
ttsEndpointPreset: "openrouter",
ttsEndpoint: undefined,
},
pets: {
installed: [builtInPet],

60
apps/desktop/src/i18n/locales/en.ts
View file

@ -8,8 +8,8 @@ export const en = {
// --- Tray menu (main process, src/tray.ts) ---
"tray.updateAvailable": "Update available: {version}...",
"tray.defaultPet": "Default Pet: {name}",
"tray.showDefaultPet": "Show Default Pet",
"tray.hideDefaultPet": "Hide Default Pet",
"tray.toggleDefaultPet.show": "Show Default Pet",
"tray.toggleDefaultPet.hide": "Hide Default Pet",
"tray.pauseAllPets": "Pause All Pets",
"tray.resumeAllPets": "Resume All Pets",
"tray.managePets": "Manage Pets...",
@ -36,7 +36,7 @@ export const en = {
"pet.status.done": "Done",
"pet.status.oops": "Oops",
"pet.status.hi": "Hi",
"pet.menu.hidePet": "Hide pet",
"pet.menu.hidePet": "Hide Pet",
"pet.menu.closePet": "Close pet",
"pet.menu.openControlCenter": "Open Control Center",
@ -270,6 +270,42 @@ export const en = {
"settings.plugins.eyebrow": "Plugin Platform",
"settings.plugins.title": "Plugin Permissions & AI",
"settings.plugins.description": "Global gates for what plugins may do. Sensitive capabilities stay off until you enable them here.",
"settings.ai.eyebrow": "AI & Chat",
"settings.ai.title": "Chat AI",
"settings.ai.description": "Model, endpoint, and system instructions used by the built-in floating chat.",
"settings.tts.eyebrow": "Voice",
"settings.tts.title": "Text-to-Speech",
"settings.tts.description": "Voice provider and controls for the pet and floating chat.",
"settings.tts.placeholder": "TTS provider settings will appear here in the next build step.",
"settings.tts.provider.title": "Provider",
"settings.tts.provider.description": "Voice engine used by the pet and floating chat.",
"settings.tts.voice.title": "Voice",
"settings.tts.voice.description": "Select a voice for the current provider.",
"settings.tts.speed.title": "Speed",
"settings.tts.speed.description": "Playback speed multiplier.",
"settings.tts.model.title": "Model",
"settings.tts.model.description": "TTS model ID. Leave blank for the provider default.",
"settings.tts.model.placeholder": "provider default",
"settings.tts.endpointPreset.title": "Endpoint preset",
"settings.tts.endpointPreset.description": "Built-in base URL preset for OpenAI-compatible providers.",
"settings.tts.endpoint.title": "Custom endpoint",
"settings.tts.endpoint.description": "Full base or speech endpoint URL when the preset is Custom.",
"settings.tts.endpoint.placeholder": "https://api.example.com/v1/audio/speech",
"settings.tts.credential.title": "API key",
"settings.tts.credential.placeholder": "Paste key",
"settings.tts.credential.save": "Save",
"settings.tts.credential.clear": "Clear",
"settings.tts.test.title": "Test voice",
"settings.tts.test.description": "Preview the configured voice with a short phrase.",
"settings.tts.test.speak": "Speak",
"settings.tts.test.stop": "Stop",
"settings.tts.toast.providerSaved": "TTS provider saved.",
"settings.tts.toast.voiceSaved": "TTS voice saved.",
"settings.tts.toast.speedSaved": "TTS speed saved.",
"settings.tts.toast.modelSaved": "TTS model saved.",
"settings.tts.toast.endpointSaved": "TTS endpoint saved.",
"settings.tts.toast.credentialSaved": "TTS credential saved.",
"settings.tts.toast.credentialCleared": "TTS credential removed.",
"settings.plugins.audio.title": "Plugins may play sound",
"settings.plugins.audio.description": "Allow plugin chimes, alerts, and bundled sounds.",
"settings.plugins.voice.title": "Plugins may speak (voice)",
@ -512,6 +548,24 @@ export const en = {
"integrations.busy.removingHooks": "Removing hooks",
"integrations.busy.updatingInstructions": "Updating instructions",
"integrations.busy.savingPath": "Saving path",
"integrations.mcpToolServers.name": "MCP Tool Servers",
"integrations.mcpToolServers.description": "Choose which built-in MCP tool servers power the floating chat.",
"integrations.mcpToolServers.status": "{count} active",
"integrations.mcpToolServers.noTools": "None active",
"integrations.mcpToolServers.builtInChat": "Built-in Chat",
"integrations.mcpToolServers.saved": "Saved {count} tool server(s).",
"integrations.openpetsMcpServer.name": "OpenPets MCP Server",
"integrations.openpetsMcpServer.description": "External-agent bridge that exposes your pet to Claude Code, Cursor, Codex CLI, and other MCP hosts.",
"integrations.openpetsMcpServer.commandModeHelp": "Choose how the OpenPets MCP server is launched for external hosts.",
"integrations.openpetsMcpServer.testServer": "Test Server",
"integrations.openpetsMcpServer.testBusy": "Testing server…",
"integrations.openpetsMcpServer.testOk": "Server responded: {output}",
"integrations.openpetsMcpServer.testError": "Server test failed: {error}",
"integrations.openpetsMcpServer.copyMcpJson": "Copy MCP JSON",
"integrations.openpetsMcpServer.status": "Ready",
"integrations.curatedToolkit.name": "Curated MCP Toolkit",
"integrations.curatedToolkit.description": "Starter stack for filesystem, Git, browser use, memory, databases, Docker, and terminal-oriented MCP lanes.",
"integrations.linkToCentralPanel": "Configure in the OpenPets MCP Server panel above.",
// --- Settings: Language section (renderer) ---
"settings.language.title": "Language",

29
apps/desktop/src/i18n/locales/es-419.ts
View file

@ -5,8 +5,8 @@ export const es419: Partial<Messages> = {
// --- Tray menu (main process, src/tray.ts) ---
"tray.updateAvailable": "Actualización disponible: {version}...",
"tray.defaultPet": "Mascota predeterminada: {name}",
"tray.showDefaultPet": "Mostrar mascota predeterminada",
"tray.hideDefaultPet": "Ocultar mascota predeterminada",
"tray.toggleDefaultPet.show": "Mostrar mascota predeterminada",
"tray.toggleDefaultPet.hide": "Ocultar mascota predeterminada",
"tray.pauseAllPets": "Pausar todas las mascotas",
"tray.resumeAllPets": "Reanudar todas las mascotas",
"tray.managePets": "Administrar mascotas...",
@ -267,6 +267,13 @@ export const es419: Partial<Messages> = {
"settings.plugins.eyebrow": "Plataforma de complementos",
"settings.plugins.title": "Permisos de complementos e IA",
"settings.plugins.description": "Controles globales de lo que pueden hacer los complementos. Las funciones sensibles permanecen desactivadas hasta que las habilites aquí.",
"settings.ai.eyebrow": "AI & Chat",
"settings.ai.title": "Chat AI",
"settings.ai.description": "Model, endpoint, and system instructions used by the built-in floating chat.",
"settings.tts.eyebrow": "Voice",
"settings.tts.title": "Text-to-Speech",
"settings.tts.description": "Voice provider and controls for the pet and floating chat.",
"settings.tts.placeholder": "TTS provider settings will appear here in the next build step.",
"settings.plugins.audio.title": "Los complementos pueden reproducir sonido",
"settings.plugins.audio.description": "Permite tonos, alertas y sonidos incluidos de los complementos.",
"settings.plugins.voice.title": "Los complementos pueden hablar (voz)",
@ -504,6 +511,24 @@ export const es419: Partial<Messages> = {
"integrations.busy.removingHooks": "Quitando hooks",
"integrations.busy.updatingInstructions": "Actualizando instrucciones",
"integrations.busy.savingPath": "Guardando ruta",
"integrations.mcpToolServers.name": "MCP Tool Servers",
"integrations.mcpToolServers.description": "Choose which built-in MCP tool servers power the floating chat.",
"integrations.mcpToolServers.status": "{count} active",
"integrations.mcpToolServers.noTools": "None active",
"integrations.mcpToolServers.builtInChat": "Built-in Chat",
"integrations.mcpToolServers.saved": "Saved {count} tool server(s).",
"integrations.openpetsMcpServer.name": "OpenPets MCP Server",
"integrations.openpetsMcpServer.description": "External-agent bridge that exposes your pet to Claude Code, Cursor, Codex CLI, and other MCP hosts.",
"integrations.openpetsMcpServer.commandModeHelp": "Choose how the OpenPets MCP server is launched for external hosts.",
"integrations.openpetsMcpServer.testServer": "Test Server",
"integrations.openpetsMcpServer.testBusy": "Testing server…",
"integrations.openpetsMcpServer.testOk": "Server responded: {output}",
"integrations.openpetsMcpServer.testError": "Server test failed: {error}",
"integrations.openpetsMcpServer.copyMcpJson": "Copy MCP JSON",
"integrations.openpetsMcpServer.status": "Ready",
"integrations.curatedToolkit.name": "Curated MCP Toolkit",
"integrations.curatedToolkit.description": "Starter stack for filesystem, Git, browser use, memory, databases, Docker, and terminal-oriented MCP lanes.",
"integrations.linkToCentralPanel": "Configure in the OpenPets MCP Server panel above.",
// --- Settings: Language section (renderer) ---
"settings.language.title": "Idioma",

29
apps/desktop/src/i18n/locales/ja.ts
View file

@ -5,8 +5,8 @@ export const ja: Partial<Messages> = {
// --- Tray menu (main process, src/tray.ts) ---
"tray.updateAvailable": "アップデートがあります: {version}...",
"tray.defaultPet": "デフォルトのペット: {name}",
"tray.showDefaultPet": "デフォルトのペットを表示",
"tray.hideDefaultPet": "デフォルトのペットを非表示",
"tray.toggleDefaultPet.show": "デフォルトのペットを表示",
"tray.toggleDefaultPet.hide": "デフォルトのペットを非表示",
"tray.pauseAllPets": "すべてのペットを一時停止",
"tray.resumeAllPets": "すべてのペットを再開",
"tray.managePets": "ペットを管理...",
@ -267,6 +267,13 @@ export const ja: Partial<Messages> = {
"settings.plugins.eyebrow": "プラグインプラットフォーム",
"settings.plugins.title": "プラグインの権限と AI",
"settings.plugins.description": "プラグインが行える操作の全体的なゲートです。機微な機能はここで有効にするまでオフのままです。",
"settings.ai.eyebrow": "AI & Chat",
"settings.ai.title": "Chat AI",
"settings.ai.description": "Model, endpoint, and system instructions used by the built-in floating chat.",
"settings.tts.eyebrow": "Voice",
"settings.tts.title": "Text-to-Speech",
"settings.tts.description": "Voice provider and controls for the pet and floating chat.",
"settings.tts.placeholder": "TTS provider settings will appear here in the next build step.",
"settings.plugins.audio.title": "プラグインによる音の再生を許可",
"settings.plugins.audio.description": "プラグインのチャイム、アラート、付属サウンドを許可します。",
"settings.plugins.voice.title": "プラグインによる音声発話を許可",
@ -504,6 +511,24 @@ export const ja: Partial<Messages> = {
"integrations.busy.removingHooks": "フックを削除中",
"integrations.busy.updatingInstructions": "手順を更新中",
"integrations.busy.savingPath": "パスを保存中",
"integrations.mcpToolServers.name": "MCP Tool Servers",
"integrations.mcpToolServers.description": "Choose which built-in MCP tool servers power the floating chat.",
"integrations.mcpToolServers.status": "{count} active",
"integrations.mcpToolServers.noTools": "None active",
"integrations.mcpToolServers.builtInChat": "Built-in Chat",
"integrations.mcpToolServers.saved": "Saved {count} tool server(s).",
"integrations.openpetsMcpServer.name": "OpenPets MCP Server",
"integrations.openpetsMcpServer.description": "External-agent bridge that exposes your pet to Claude Code, Cursor, Codex CLI, and other MCP hosts.",
"integrations.openpetsMcpServer.commandModeHelp": "Choose how the OpenPets MCP server is launched for external hosts.",
"integrations.openpetsMcpServer.testServer": "Test Server",
"integrations.openpetsMcpServer.testBusy": "Testing server…",
"integrations.openpetsMcpServer.testOk": "Server responded: {output}",
"integrations.openpetsMcpServer.testError": "Server test failed: {error}",
"integrations.openpetsMcpServer.copyMcpJson": "Copy MCP JSON",
"integrations.openpetsMcpServer.status": "Ready",
"integrations.curatedToolkit.name": "Curated MCP Toolkit",
"integrations.curatedToolkit.description": "Starter stack for filesystem, Git, browser use, memory, databases, Docker, and terminal-oriented MCP lanes.",
"integrations.linkToCentralPanel": "Configure in the OpenPets MCP Server panel above.",
// --- Settings: Language section (renderer) ---
"settings.language.title": "言語",

29
apps/desktop/src/i18n/locales/ko.ts
View file

@ -5,8 +5,8 @@ export const ko: Partial<Messages> = {
// --- Tray menu (main process, src/tray.ts) ---
"tray.updateAvailable": "업데이트 사용 가능: {version}...",
"tray.defaultPet": "기본 펫: {name}",
"tray.showDefaultPet": "기본 펫 표시",
"tray.hideDefaultPet": "기본 펫 숨기기",
"tray.toggleDefaultPet.show": "기본 펫 표시",
"tray.toggleDefaultPet.hide": "기본 펫 숨기기",
"tray.pauseAllPets": "모든 펫 일시정지",
"tray.resumeAllPets": "모든 펫 재개",
"tray.managePets": "펫 관리...",
@ -267,6 +267,13 @@ export const ko: Partial<Messages> = {
"settings.plugins.eyebrow": "플러그인 플랫폼",
"settings.plugins.title": "플러그인 권한 및 AI",
"settings.plugins.description": "플러그인이 할 수 있는 작업에 대한 전역 제어입니다. 민감한 기능은 여기에서 활성화하기 전까지 꺼져 있습니다.",
"settings.ai.eyebrow": "AI & Chat",
"settings.ai.title": "Chat AI",
"settings.ai.description": "Model, endpoint, and system instructions used by the built-in floating chat.",
"settings.tts.eyebrow": "Voice",
"settings.tts.title": "Text-to-Speech",
"settings.tts.description": "Voice provider and controls for the pet and floating chat.",
"settings.tts.placeholder": "TTS provider settings will appear here in the next build step.",
"settings.plugins.audio.title": "플러그인 소리 재생 허용",
"settings.plugins.audio.description": "플러그인 알림음, 경고음, 번들 사운드를 허용합니다.",
"settings.plugins.voice.title": "플러그인 음성 말하기 허용",
@ -504,6 +511,24 @@ export const ko: Partial<Messages> = {
"integrations.busy.removingHooks": "훅 제거 중",
"integrations.busy.updatingInstructions": "지침 업데이트 중",
"integrations.busy.savingPath": "경로 저장 중",
"integrations.mcpToolServers.name": "MCP Tool Servers",
"integrations.mcpToolServers.description": "Choose which built-in MCP tool servers power the floating chat.",
"integrations.mcpToolServers.status": "{count} active",
"integrations.mcpToolServers.noTools": "None active",
"integrations.mcpToolServers.builtInChat": "Built-in Chat",
"integrations.mcpToolServers.saved": "Saved {count} tool server(s).",
"integrations.openpetsMcpServer.name": "OpenPets MCP Server",
"integrations.openpetsMcpServer.description": "External-agent bridge that exposes your pet to Claude Code, Cursor, Codex CLI, and other MCP hosts.",
"integrations.openpetsMcpServer.commandModeHelp": "Choose how the OpenPets MCP server is launched for external hosts.",
"integrations.openpetsMcpServer.testServer": "Test Server",
"integrations.openpetsMcpServer.testBusy": "Testing server…",
"integrations.openpetsMcpServer.testOk": "Server responded: {output}",
"integrations.openpetsMcpServer.testError": "Server test failed: {error}",
"integrations.openpetsMcpServer.copyMcpJson": "Copy MCP JSON",
"integrations.openpetsMcpServer.status": "Ready",
"integrations.curatedToolkit.name": "Curated MCP Toolkit",
"integrations.curatedToolkit.description": "Starter stack for filesystem, Git, browser use, memory, databases, Docker, and terminal-oriented MCP lanes.",
"integrations.linkToCentralPanel": "Configure in the OpenPets MCP Server panel above.",
// --- Settings: Language section (renderer) ---
"settings.language.title": "언어",

29
apps/desktop/src/i18n/locales/pt-BR.ts
View file

@ -5,8 +5,8 @@ export const ptBR: Partial<Messages> = {
// --- Tray menu (main process, src/tray.ts) ---
"tray.updateAvailable": "Atualização disponível: {version}...",
"tray.defaultPet": "Pet padrão: {name}",
"tray.showDefaultPet": "Mostrar pet padrão",
"tray.hideDefaultPet": "Ocultar pet padrão",
"tray.toggleDefaultPet.show": "Mostrar pet padrão",
"tray.toggleDefaultPet.hide": "Ocultar pet padrão",
"tray.pauseAllPets": "Pausar todos os pets",
"tray.resumeAllPets": "Retomar todos os pets",
"tray.managePets": "Gerenciar pets...",
@ -267,6 +267,13 @@ export const ptBR: Partial<Messages> = {
"settings.plugins.eyebrow": "Plataforma de plugins",
"settings.plugins.title": "Permissões de plugins e IA",
"settings.plugins.description": "Controles globais do que os plugins podem fazer. Recursos sensíveis ficam desativados até que você os ative aqui.",
"settings.ai.eyebrow": "AI & Chat",
"settings.ai.title": "Chat AI",
"settings.ai.description": "Model, endpoint, and system instructions used by the built-in floating chat.",
"settings.tts.eyebrow": "Voice",
"settings.tts.title": "Text-to-Speech",
"settings.tts.description": "Voice provider and controls for the pet and floating chat.",
"settings.tts.placeholder": "TTS provider settings will appear here in the next build step.",
"settings.plugins.audio.title": "Plugins podem reproduzir som",
"settings.plugins.audio.description": "Permitir sinos, alertas e sons incluídos dos plugins.",
"settings.plugins.voice.title": "Plugins podem falar (voz)",
@ -504,6 +511,24 @@ export const ptBR: Partial<Messages> = {
"integrations.busy.removingHooks": "Removendo hooks",
"integrations.busy.updatingInstructions": "Atualizando instruções",
"integrations.busy.savingPath": "Salvando caminho",
"integrations.mcpToolServers.name": "MCP Tool Servers",
"integrations.mcpToolServers.description": "Choose which built-in MCP tool servers power the floating chat.",
"integrations.mcpToolServers.status": "{count} active",
"integrations.mcpToolServers.noTools": "None active",
"integrations.mcpToolServers.builtInChat": "Built-in Chat",
"integrations.mcpToolServers.saved": "Saved {count} tool server(s).",
"integrations.openpetsMcpServer.name": "OpenPets MCP Server",
"integrations.openpetsMcpServer.description": "External-agent bridge that exposes your pet to Claude Code, Cursor, Codex CLI, and other MCP hosts.",
"integrations.openpetsMcpServer.commandModeHelp": "Choose how the OpenPets MCP server is launched for external hosts.",
"integrations.openpetsMcpServer.testServer": "Test Server",
"integrations.openpetsMcpServer.testBusy": "Testing server…",
"integrations.openpetsMcpServer.testOk": "Server responded: {output}",
"integrations.openpetsMcpServer.testError": "Server test failed: {error}",
"integrations.openpetsMcpServer.copyMcpJson": "Copy MCP JSON",
"integrations.openpetsMcpServer.status": "Ready",
"integrations.curatedToolkit.name": "Curated MCP Toolkit",
"integrations.curatedToolkit.description": "Starter stack for filesystem, Git, browser use, memory, databases, Docker, and terminal-oriented MCP lanes.",
"integrations.linkToCentralPanel": "Configure in the OpenPets MCP Server panel above.",
// --- Settings: Language section (renderer) ---
"settings.language.title": "Idioma",

29
apps/desktop/src/i18n/locales/zh-Hans.ts
View file

@ -5,8 +5,8 @@ export const zhHans: Partial<Messages> = {
// --- Tray menu (main process, src/tray.ts) ---
"tray.updateAvailable": "有可用更新：{version}...",
"tray.defaultPet": "默认宠物：{name}",
"tray.showDefaultPet": "显示默认宠物",
"tray.hideDefaultPet": "隐藏默认宠物",
"tray.toggleDefaultPet.show": "显示默认宠物",
"tray.toggleDefaultPet.hide": "隐藏默认宠物",
"tray.pauseAllPets": "暂停所有宠物",
"tray.resumeAllPets": "恢复所有宠物",
"tray.managePets": "管理宠物...",
@ -267,6 +267,13 @@ export const zhHans: Partial<Messages> = {
"settings.plugins.eyebrow": "插件平台",
"settings.plugins.title": "插件权限与 AI",
"settings.plugins.description": "插件行为的全局开关。敏感能力在你于此启用前保持关闭。",
"settings.ai.eyebrow": "AI & Chat",
"settings.ai.title": "Chat AI",
"settings.ai.description": "Model, endpoint, and system instructions used by the built-in floating chat.",
"settings.tts.eyebrow": "Voice",
"settings.tts.title": "Text-to-Speech",
"settings.tts.description": "Voice provider and controls for the pet and floating chat.",
"settings.tts.placeholder": "TTS provider settings will appear here in the next build step.",
"settings.plugins.audio.title": "允许插件播放声音",
"settings.plugins.audio.description": "允许插件播放提示音、警报和内置音效。",
"settings.plugins.voice.title": "允许插件说话（语音）",
@ -504,6 +511,24 @@ export const zhHans: Partial<Messages> = {
"integrations.busy.removingHooks": "移除钩子中",
"integrations.busy.updatingInstructions": "更新说明中",
"integrations.busy.savingPath": "保存路径中",
"integrations.mcpToolServers.name": "MCP Tool Servers",
"integrations.mcpToolServers.description": "Choose which built-in MCP tool servers power the floating chat.",
"integrations.mcpToolServers.status": "{count} active",
"integrations.mcpToolServers.noTools": "None active",
"integrations.mcpToolServers.builtInChat": "Built-in Chat",
"integrations.mcpToolServers.saved": "Saved {count} tool server(s).",
"integrations.openpetsMcpServer.name": "OpenPets MCP Server",
"integrations.openpetsMcpServer.description": "External-agent bridge that exposes your pet to Claude Code, Cursor, Codex CLI, and other MCP hosts.",
"integrations.openpetsMcpServer.commandModeHelp": "Choose how the OpenPets MCP server is launched for external hosts.",
"integrations.openpetsMcpServer.testServer": "Test Server",
"integrations.openpetsMcpServer.testBusy": "Testing server…",
"integrations.openpetsMcpServer.testOk": "Server responded: {output}",
"integrations.openpetsMcpServer.testError": "Server test failed: {error}",
"integrations.openpetsMcpServer.copyMcpJson": "Copy MCP JSON",
"integrations.openpetsMcpServer.status": "Ready",
"integrations.curatedToolkit.name": "Curated MCP Toolkit",
"integrations.curatedToolkit.description": "Starter stack for filesystem, Git, browser use, memory, databases, Docker, and terminal-oriented MCP lanes.",
"integrations.linkToCentralPanel": "Configure in the OpenPets MCP Server panel above.",
// --- Settings: Language section (renderer) ---
"settings.language.title": "语言",

29
apps/desktop/src/i18n/locales/zh-Hant.ts
View file

@ -5,8 +5,8 @@ export const zhHant: Partial<Messages> = {
// --- Tray menu (main process, src/tray.ts) ---
"tray.updateAvailable": "有可用更新：{version}...",
"tray.defaultPet": "預設寵物：{name}",
"tray.showDefaultPet": "顯示預設寵物",
"tray.hideDefaultPet": "隱藏預設寵物",
"tray.toggleDefaultPet.show": "顯示預設寵物",
"tray.toggleDefaultPet.hide": "隱藏預設寵物",
"tray.pauseAllPets": "暫停所有寵物",
"tray.resumeAllPets": "恢復所有寵物",
"tray.managePets": "管理寵物...",
@ -267,6 +267,13 @@ export const zhHant: Partial<Messages> = {
"settings.plugins.eyebrow": "外掛平台",
"settings.plugins.title": "外掛權限與 AI",
"settings.plugins.description": "控管外掛可執行行為的全域開關。敏感功能在你於此啟用前都會保持關閉。",
"settings.ai.eyebrow": "AI & Chat",
"settings.ai.title": "Chat AI",
"settings.ai.description": "Model, endpoint, and system instructions used by the built-in floating chat.",
"settings.tts.eyebrow": "Voice",
"settings.tts.title": "Text-to-Speech",
"settings.tts.description": "Voice provider and controls for the pet and floating chat.",
"settings.tts.placeholder": "TTS provider settings will appear here in the next build step.",
"settings.plugins.audio.title": "允許外掛播放聲音",
"settings.plugins.audio.description": "允許外掛的提示音、警示音與內建音效。",
"settings.plugins.voice.title": "允許外掛說話（語音）",
@ -504,6 +511,24 @@ export const zhHant: Partial<Messages> = {
"integrations.busy.removingHooks": "移除 Hooks 中",
"integrations.busy.updatingInstructions": "更新指示中",
"integrations.busy.savingPath": "儲存路徑中",
"integrations.mcpToolServers.name": "MCP Tool Servers",
"integrations.mcpToolServers.description": "Choose which built-in MCP tool servers power the floating chat.",
"integrations.mcpToolServers.status": "{count} active",
"integrations.mcpToolServers.noTools": "None active",
"integrations.mcpToolServers.builtInChat": "Built-in Chat",
"integrations.mcpToolServers.saved": "Saved {count} tool server(s).",
"integrations.openpetsMcpServer.name": "OpenPets MCP Server",
"integrations.openpetsMcpServer.description": "External-agent bridge that exposes your pet to Claude Code, Cursor, Codex CLI, and other MCP hosts.",
"integrations.openpetsMcpServer.commandModeHelp": "Choose how the OpenPets MCP server is launched for external hosts.",
"integrations.openpetsMcpServer.testServer": "Test Server",
"integrations.openpetsMcpServer.testBusy": "Testing server…",
"integrations.openpetsMcpServer.testOk": "Server responded: {output}",
"integrations.openpetsMcpServer.testError": "Server test failed: {error}",
"integrations.openpetsMcpServer.copyMcpJson": "Copy MCP JSON",
"integrations.openpetsMcpServer.status": "Ready",
"integrations.curatedToolkit.name": "Curated MCP Toolkit",
"integrations.curatedToolkit.description": "Starter stack for filesystem, Git, browser use, memory, databases, Docker, and terminal-oriented MCP lanes.",
"integrations.linkToCentralPanel": "Configure in the OpenPets MCP Server panel above.",
// --- Settings: Language section (renderer) ---
"settings.language.title": "語言",

2
apps/desktop/src/logger.ts
View file

@ -4,7 +4,7 @@ import { appendFile } from "node:fs/promises";
import { dirname, join } from "node:path";
export type LogLevel = "debug" | "info" | "warn" | "error";
export type LogScope = "app" | "ipc" | "lease" | "pet.default" | "pet.agent" | "pet.window" | "plugin" | "state" | "tray" | "ui";
export type LogScope = "app" | "ipc" | "lease" | "pet.default" | "pet.agent" | "pet.window" | "plugin" | "state" | "tray" | "tts" | "ui";
type LogFields = Record<string, unknown>;

9
apps/desktop/src/openapi-chat.ts
View file

@ -9,6 +9,7 @@ import type { OpenPetsReaction } from "./local-ipc-protocol.js";
import { buildRelevantKnowledgeContext } from "./knowledge-store.js";
import { buildRelevantMemoryContext, capturePromptMemories } from "./openpets-memory.js";
import { getMcpChatClientManager, type McpChatToolDefinition } from "./mcp-chat-client.js";
import { speakTts } from "./tts-service.js";
type OpenApiChatTransport = "responses" | "chat-completions";
@ -294,6 +295,12 @@ export async function sendOpenApiChatPrompt(prompt: string): Promise<OpenApiChat
return sendOpenApiChatPromptPlain(trimmedPrompt);
}
function speakAssistantResponse(text: string): void {
void speakTts(text, "prompt").catch((error: unknown) => {
warn("app", "TTS for assistant response failed", { error: error instanceof Error ? error.message : String(error) });
});
}
async function sendOpenApiChatPromptPlain(prompt: string): Promise<OpenApiChatPromptResult> {
const credential = getRequiredCredential();
const model = getConfiguredOpenApiChatModel();
@ -385,6 +392,7 @@ async function sendOpenApiChatPromptPlain(prompt: string): Promise<OpenApiChatPr
const responseId = parsed.responseId ?? `response-${Date.now()}-${Math.random().toString(36).slice(2, 8)}`;
previousResponseId = attempt.transport === "responses" ? parsed.responseId : undefined;
appendTranscriptEntry("assistant", parsed.text);
speakAssistantResponse(parsed.text);
showOpenApiSuccessState(parsed.text);
info("app", "openapi prompt completed", {
model,
@ -1019,6 +1027,7 @@ async function sendOpenApiChatPromptWithTools(prompt: string): Promise<OpenApiCh
}
appendTranscriptEntry("assistant", finalText);
speakAssistantResponse(finalText);
showOpenApiSuccessState(finalText);
info("app", "openapi prompt with tools completed", {
model,

12
apps/desktop/src/plugin-voice.ts
View file

@ -1,8 +1,7 @@
import { BrowserWindow, session } from "electron";
import { getDefaultPetWindowForPlugins } from "./default-pet-controller.js";
import { debug } from "./logger.js";
import { speakPetWindowTts, stopPetWindowTts } from "./pet-window.js";
import { speakTts, stopTts } from "./tts-service.js";
import type { PluginAiGateway } from "./plugin-ai-gateway.js";
/**
@ -13,15 +12,12 @@ import type { PluginAiGateway } from "./plugin-ai-gateway.js";
* the user's configured AI provider. Never ambient.
*/
export async function pluginVoiceSpeak(text: string, opts: { voice?: string; rate?: number }): Promise<void> {
const window = getDefaultPetWindowForPlugins();
if (!window) throw new Error("No pet window is available for speech.");
speakPetWindowTts(window, text, opts);
export async function pluginVoiceSpeak(text: string, _opts: { voice?: string; rate?: number }): Promise<void> {
await speakTts(text, "pet");
}
export function pluginVoiceStop(): void {
const window = getDefaultPetWindowForPlugins();
if (window) stopPetWindowTts(window);
stopTts("pet");
}
let listenInProgress = false;

4
apps/desktop/src/prompt-window.ts
View file

@ -138,6 +138,10 @@ export function installPromptWindowHandlers(): void {
});
}
export function getPromptWindow(): BrowserWindow | null {
return promptWindow && !promptWindow.isDestroyed() ? promptWindow : null;
}
export function openPromptWindow(): void {
const existing = promptWindow;
if (existing && !existing.isDestroyed()) {

1019
apps/desktop/src/renderer/src/main.tsx
View file

File diff suppressed because it is too large Load diff

2
apps/desktop/src/tray.ts
View file

@ -47,7 +47,7 @@ export function refreshTrayMenu(): void {
click: () => openControlCenterWindow("pets"),
},
{
label: isDefaultPetVisible() ? t("tray.hideDefaultPet") : t("tray.showDefaultPet"),
label: isDefaultPetVisible() ? t("tray.toggleDefaultPet.hide") : t("tray.toggleDefaultPet.show"),
click: () => {
if (isDefaultPetVisible()) {
hideDefaultPet();

56
apps/desktop/src/tts-catalog.ts Normal file
View file

@ -0,0 +1,56 @@
export type TtsProviderId = "system" | "openai" | "elevenlabs" | "piper" | "openai-compatible";
export type TtsEndpointPreset = "openrouter" | "litellm" | "wavespeedai" | "custom";
export interface TtsVoice {
readonly id: string;
readonly label: string;
}
export interface TtsProviderDescriptor {
readonly id: TtsProviderId;
readonly label: string;
readonly defaultModel: string;
readonly defaultVoice: string;
readonly voices: readonly TtsVoice[];
}
const openAiVoices: TtsVoice[] = [
{ id: "alloy", label: "Alloy" },
{ id: "echo", label: "Echo" },
{ id: "fable", label: "Fable" },
{ id: "onyx", label: "Onyx" },
{ id: "nova", label: "Nova" },
{ id: "shimmer", label: "Shimmer" },
];
export function listTtsProviders(): TtsProviderDescriptor[] {
return [
{ id: "system", label: "System voice", defaultModel: "", defaultVoice: "", voices: [] },
{ id: "openai", label: "OpenAI TTS", defaultModel: "tts-1", defaultVoice: "alloy", voices: openAiVoices },
{ id: "elevenlabs", label: "ElevenLabs", defaultModel: "eleven_multilingual_v2", defaultVoice: "", voices: [] },
{ id: "piper", label: "Piper (local)", defaultModel: "", defaultVoice: "", voices: [] },
{ id: "openai-compatible", label: "OpenAI-compatible", defaultModel: "", defaultVoice: "", voices: openAiVoices },
];
}
export function getTtsProviderDefaults(provider: TtsProviderId): { model: string; voice: string } {
const entry = listTtsProviders().find((p) => p.id === provider);
return { model: entry?.defaultModel ?? "", voice: entry?.defaultVoice ?? "" };
}
export function getTtsEndpointPreset(preset: TtsEndpointPreset): { endpoint: string; model: string } {
switch (preset) {
case "openrouter":
return { endpoint: "https://openrouter.ai/api/v1", model: "" };
case "litellm":
return { endpoint: "http://localhost:4000/v1", model: "" };
case "wavespeedai":
return { endpoint: "https://api.wavespeed.ai/v1", model: "" };
case "custom":
return { endpoint: "", model: "" };
}
}
export function getOpenAiTtsVoices(): readonly TtsVoice[] {
return openAiVoices;
}

109
apps/desktop/src/tts-credentials.ts Normal file
View file

@ -0,0 +1,109 @@
import { app, safeStorage } from "electron";
import { existsSync, mkdirSync, readFileSync, writeFileSync } from "node:fs";
import { dirname, join } from "node:path";
import { debug, error as logError, info } from "./logger.js";
export interface TtsCredentialStatus {
readonly hasCredential: boolean;
readonly storageMode: "encrypted" | "plain";
readonly provider: string;
}
interface StoredTtsCredentialV1 {
readonly version: 1;
readonly provider: string;
readonly storageMode: "encrypted" | "plain";
readonly apiKey: string;
}
const ttsCredentialFileName = "openpets-tts-credential.json";
export function getTtsCredentialStatus(provider: string): TtsCredentialStatus {
const stored = readStoredCredential();
return {
hasCredential: stored?.provider === provider && Boolean(stored.apiKey),
storageMode: stored?.storageMode ?? getPreferredStorageMode(),
provider: stored?.provider ?? provider,
};
}
export function getTtsCredential(provider: string): string | undefined {
const stored = readStoredCredential();
if (!stored || stored.provider !== provider || !stored.apiKey) return undefined;
return getStoredCredential(stored);
}
export function saveTtsCredential(provider: string, credential: string): TtsCredentialStatus {
const trimmed = credential.trim();
if (!trimmed) {
throw new Error("API key or token cannot be empty.");
}
if (trimmed.length > 2_048 || /[\0\r\n]/.test(trimmed)) {
throw new Error("API key or token is not valid.");
}
const storageMode = getPreferredStorageMode();
const serialized = storageMode === "encrypted"
? safeStorage.encryptString(trimmed).toString("base64")
: trimmed;
writeStoredCredential({ version: 1, provider, storageMode, apiKey: serialized });
info("app", "TTS credential saved", { provider, storageMode });
return getTtsCredentialStatus(provider);
}
export function clearTtsCredential(provider: string): TtsCredentialStatus {
const stored = readStoredCredential();
writeStoredCredential({ version: 1, provider, storageMode: getPreferredStorageMode(), apiKey: "" });
info("app", "TTS credential cleared", { provider });
return { hasCredential: false, storageMode: stored?.storageMode ?? getPreferredStorageMode(), provider };
}
function getPreferredStorageMode(): "encrypted" | "plain" {
return safeStorage.isEncryptionAvailable() ? "encrypted" : "plain";
}
function getCredentialPath(): string {
return join(app.getPath("userData"), ttsCredentialFileName);
}
function readStoredCredential(): StoredTtsCredentialV1 | null {
const path = getCredentialPath();
if (!existsSync(path)) return null;
try {
const raw = JSON.parse(readFileSync(path, "utf8")) as unknown;
if (!isRecord(raw) || raw.version !== 1) return null;
const provider = typeof raw.provider === "string" ? raw.provider : "";
const storageMode = raw.storageMode === "encrypted" || raw.storageMode === "plain" ? raw.storageMode : "plain";
const apiKey = typeof raw.apiKey === "string" ? raw.apiKey : "";
return { version: 1, provider, storageMode, apiKey };
} catch (error) {
logError("tts", "Failed to read stored TTS credential", { error: String(error) });
return null;
}
}
function writeStoredCredential(stored: StoredTtsCredentialV1): void {
const path = getCredentialPath();
const dir = dirname(path);
if (!existsSync(dir)) {
mkdirSync(dir, { recursive: true });
}
writeFileSync(path, JSON.stringify(stored, null, 2), "utf8");
}
function getStoredCredential(stored: StoredTtsCredentialV1): string | undefined {
if (!stored.apiKey) return undefined;
if (stored.storageMode === "encrypted") {
if (!safeStorage.isEncryptionAvailable()) {
throw new Error("Encrypted TTS credentials are unavailable on this machine right now. Re-save the key in Settings.");
}
return safeStorage.decryptString(Buffer.from(stored.apiKey, "base64"));
}
return stored.apiKey;
}
function isRecord(value: unknown): value is Record<string, unknown> {
return typeof value === "object" && value !== null && !Array.isArray(value);
}

256
apps/desktop/src/tts-engine.ts Normal file
View file

@ -0,0 +1,256 @@
import { spawn } from "node:child_process";
import { existsSync, readFileSync, statSync } from "node:fs";
import { type OpenPetsStateV1 } from "./app-state.js";
import { getTtsCredential } from "./tts-credentials.js";
import { getOpenAiTtsVoices, getTtsEndpointPreset, getTtsProviderDefaults, listTtsProviders, type TtsEndpointPreset, type TtsProviderId, type TtsVoice } from "./tts-catalog.js";
export type { TtsEndpointPreset, TtsProviderId, TtsVoice };
export { getTtsEndpointPreset, getTtsProviderDefaults, listTtsProviders } from "./tts-catalog.js";
export interface TtsSynthesizeResult {
readonly provider: TtsProviderId;
readonly audio?: Buffer;
readonly mimeType?: string;
}
export interface TtsSynthesizeOptions {
readonly text: string;
readonly provider: TtsProviderId;
readonly voice?: string;
readonly speed?: number;
readonly model?: string;
readonly endpointPreset?: TtsEndpointPreset;
readonly endpoint?: string;
}
export async function synthesizeTts(opts: TtsSynthesizeOptions): Promise<TtsSynthesizeResult> {
const text = (opts.text ?? "").trim();
if (!text) throw new TtsError("No text to speak.");
if (text.length > 2000) {
throw new TtsError("Text is too long. TTS is limited to 2000 characters.");
}
if (opts.provider === "system") {
return { provider: "system" };
}
const credential = getTtsCredential(opts.provider);
if (!credential && opts.provider !== "piper") {
throw new TtsError(`A credential is required for ${getProviderLabel(opts.provider)}.`);
}
switch (opts.provider) {
case "openai":
return synthesizeOpenAi(text, opts, credential ?? "");
case "elevenlabs":
return synthesizeElevenLabs(text, opts, credential ?? "");
case "piper":
return synthesizePiper(text, opts);
case "openai-compatible":
return synthesizeOpenAiCompatible(text, opts, credential ?? "");
default:
throw new TtsError(`Unsupported TTS provider: ${opts.provider}`);
}
}
export async function fetchTtsVoices(provider: TtsProviderId, credential?: string): Promise<TtsVoice[]> {
if (provider === "openai" || provider === "openai-compatible") return [...getOpenAiTtsVoices()];
if (provider === "system" || provider === "piper") return [];
if (provider === "elevenlabs") {
if (!credential) return [];
return fetchElevenLabsVoices(credential);
}
return [];
}
export class TtsError extends Error {
constructor(message: string) {
super(message);
this.name = "TtsError";
}
}
function getProviderLabel(provider: TtsProviderId): string {
return listTtsProviders().find((p) => p.id === provider)?.label ?? provider;
}
function assertHttpsUrl(url: string, allowLocalhost: boolean): void {
let parsed: URL;
try {
parsed = new URL(url);
} catch {
throw new TtsError("Invalid endpoint URL.");
}
if (parsed.protocol !== "https:" && !(allowLocalhost && parsed.protocol === "http:")) {
throw new TtsError("Endpoint must use HTTPS (or HTTP localhost for local proxies).");
}
if (parsed.protocol === "https:" && parsed.hostname === "localhost") {
throw new TtsError("HTTPS localhost is not supported. Use http://localhost for local proxies.");
}
if (!allowLocalhost && (parsed.hostname === "localhost" || parsed.hostname === "127.0.0.1" || parsed.hostname.startsWith("192.168.") || parsed.hostname.startsWith("10."))) {
throw new TtsError("Cloud providers cannot use localhost or private IP endpoints.");
}
}
function resolveEndpoint(opts: TtsSynthesizeOptions, defaultEndpoint: string, allowLocalhost: boolean): string {
let endpoint = opts.endpoint?.trim() || defaultEndpoint;
if (opts.provider === "openai-compatible" && opts.endpointPreset && opts.endpointPreset !== "custom") {
endpoint = getTtsEndpointPreset(opts.endpointPreset).endpoint;
}
endpoint = endpoint.replace(/\/$/, "");
assertHttpsUrl(endpoint, allowLocalhost);
return endpoint;
}
async function synthesizeOpenAi(text: string, opts: TtsSynthesizeOptions, apiKey: string): Promise<TtsSynthesizeResult> {
const endpoint = resolveEndpoint(opts, "https://api.openai.com/v1", false);
const model = opts.model?.trim() || "tts-1";
const voice = opts.voice?.trim() || "alloy";
const speed = typeof opts.speed === "number" ? Math.max(0.5, Math.min(2, opts.speed)) : 1;
const response = await fetch(`${endpoint}/audio/speech`, {
method: "POST",
headers: {
"Authorization": `Bearer ${apiKey}`,
"Content-Type": "application/json",
},
body: JSON.stringify({ model, voice, input: text, speed }),
signal: AbortSignal.timeout(30_000),
});
if (!response.ok) {
const body = await response.text().catch(() => "");
throw new TtsError(`OpenAI TTS failed (${response.status}): ${body.slice(0, 200)}`);
}
const arrayBuffer = await response.arrayBuffer();
const buffer = Buffer.from(arrayBuffer);
validateAudioBuffer(buffer, "mp3");
return { provider: "openai", audio: buffer, mimeType: "audio/mpeg" };
}
async function synthesizeOpenAiCompatible(text: string, opts: TtsSynthesizeOptions, apiKey: string): Promise<TtsSynthesizeResult> {
const preset = opts.endpointPreset ?? "custom";
const allowLocalhost = preset === "litellm" || preset === "custom";
const endpoint = resolveEndpoint(opts, "", allowLocalhost);
if (!endpoint) throw new TtsError("A custom endpoint is required for OpenAI-compatible TTS.");
const model = opts.model?.trim() || "";
const voice = opts.voice?.trim() || "alloy";
const speed = typeof opts.speed === "number" ? Math.max(0.5, Math.min(2, opts.speed)) : 1;
const response = await fetch(`${endpoint}/audio/speech`, {
method: "POST",
headers: {
"Authorization": `Bearer ${apiKey}`,
"Content-Type": "application/json",
},
body: JSON.stringify({ model, voice, input: text, speed }),
signal: AbortSignal.timeout(30_000),
});
if (!response.ok) {
const body = await response.text().catch(() => "");
throw new TtsError(`TTS endpoint failed (${response.status}): ${body.slice(0, 200)}`);
}
const arrayBuffer = await response.arrayBuffer();
const buffer = Buffer.from(arrayBuffer);
validateAudioBuffer(buffer, "mp3");
return { provider: "openai-compatible", audio: buffer, mimeType: "audio/mpeg" };
}
async function synthesizeElevenLabs(text: string, opts: TtsSynthesizeOptions, apiKey: string): Promise<TtsSynthesizeResult> {
const voiceId = opts.voice?.trim();
if (!voiceId) throw new TtsError("A voice ID is required for ElevenLabs.");
const model = opts.model?.trim() || "eleven_multilingual_v2";
const response = await fetch(`https://api.elevenlabs.io/v1/text-to-speech/${encodeURIComponent(voiceId)}/stream`, {
method: "POST",
headers: {
"xi-api-key": apiKey,
"Content-Type": "application/json",
"Accept": "audio/mpeg",
},
body: JSON.stringify({
text,
model_id: model,
voice_settings: { stability: 0.5, similarity_boost: 0.75 },
}),
signal: AbortSignal.timeout(30_000),
});
if (!response.ok) {
const body = await response.text().catch(() => "");
throw new TtsError(`ElevenLabs TTS failed (${response.status}): ${body.slice(0, 200)}`);
}
const arrayBuffer = await response.arrayBuffer();
const buffer = Buffer.from(arrayBuffer);
validateAudioBuffer(buffer, "mp3");
return { provider: "elevenlabs", audio: buffer, mimeType: "audio/mpeg" };
}
async function fetchElevenLabsVoices(apiKey: string): Promise<TtsVoice[]> {
const response = await fetch("https://api.elevenlabs.io/v1/voices", {
headers: { "xi-api-key": apiKey },
signal: AbortSignal.timeout(15_000),
});
if (!response.ok) return [];
const data = await response.json() as unknown;
if (!data || typeof data !== "object" || !Array.isArray((data as Record<string, unknown>).voices)) return [];
return ((data as { voices: Array<{ voice_id?: string; name?: string }> }).voices ?? [])
.filter((voice): voice is { voice_id: string; name?: string } => typeof voice.voice_id === "string")
.map((voice) => ({ id: voice.voice_id, label: voice.name || voice.voice_id }));
}
function synthesizePiper(text: string, opts: TtsSynthesizeOptions): Promise<TtsSynthesizeResult> {
const modelPath = opts.voice?.trim() || opts.model?.trim();
if (!modelPath) throw new TtsError("A Piper model path is required.");
if (!existsSync(modelPath)) throw new TtsError(`Piper model not found: ${modelPath}`);
const stat = statSync(modelPath);
if (!stat.isFile()) throw new TtsError(`Piper model path is not a file: ${modelPath}`);
return new Promise((resolve, reject) => {
const child = spawn("piper", ["--model", modelPath, "--output_file", "-"], {
timeout: 30_000,
});
const chunks: Buffer[] = [];
let stderr = "";
child.stdout.on("data", (chunk) => chunks.push(Buffer.isBuffer(chunk) ? chunk : Buffer.from(chunk)));
child.stderr.on("data", (chunk) => { stderr += String(chunk); });
child.on("error", (error) => reject(new TtsError(`Failed to run piper: ${error.message}`)));
child.on("close", (code) => {
if (code !== 0) {
reject(new TtsError(`Piper exited with code ${code ?? "unknown"}: ${stderr.slice(0, 200)}`));
return;
}
const buffer = Buffer.concat(chunks);
try {
validateAudioBuffer(buffer, "wav");
} catch (err) {
reject(err);
return;
}
resolve({ provider: "piper", audio: buffer, mimeType: "audio/wav" });
});
child.stdin.write(text, "utf8");
child.stdin.end();
});
}
function validateAudioBuffer(buffer: Buffer, expectedFormat: "mp3" | "wav"): void {
if (buffer.length < 8) {
throw new TtsError("TTS returned an empty or truncated audio response.");
}
if (expectedFormat === "wav" && buffer.toString("ascii", 0, 4) !== "RIFF") {
throw new TtsError("TTS did not return a valid WAV file.");
}
if (expectedFormat === "mp3") {
const header = buffer.subarray(0, 4).toString("hex");
const isMp3 = header.startsWith("494433") || header.startsWith("fffb") || header.startsWith("fff3") || header.startsWith("fff2") || header.startsWith("52494646");
if (!isMp3) {
throw new TtsError("TTS did not return a valid MP3 audio response.");
}
}
}

111
apps/desktop/src/tts-service.ts Normal file
View file

@ -0,0 +1,111 @@
import type { BrowserWindow } from "electron";
import { getAppStateSnapshot } from "./app-state.js";
import { getDefaultPetWindowForPlugins } from "./default-pet-controller.js";
import { isInQuietHours } from "./plugin-platform-settings.js";
import { getPromptWindow } from "./prompt-window.js";
import { debug, error as logError } from "./logger.js";
import { getTtsCredential, getTtsCredentialStatus } from "./tts-credentials.js";
import { fetchTtsVoices, listTtsProviders, type TtsProviderId, type TtsSynthesizeOptions, type TtsVoice, synthesizeTts } from "./tts-engine.js";
export { TtsProviderId, TtsSynthesizeOptions, TtsVoice };
export interface TtsSettingsSnapshot {
readonly provider: TtsProviderId;
readonly voice: string;
readonly speed: number;
readonly model: string;
readonly endpointPreset: "openrouter" | "litellm" | "wavespeedai" | "custom";
readonly endpoint: string;
readonly hasCredential: boolean;
readonly providers: { readonly id: TtsProviderId; readonly label: string; readonly defaultModel: string; readonly defaultVoice: string; readonly voices: readonly TtsVoice[] }[];
}
export function getTtsSettingsSnapshot(): TtsSettingsSnapshot {
const prefs = getAppStateSnapshot().preferences;
const provider = prefs.ttsProvider ?? "system";
const defaults = listTtsProviders().find((p) => p.id === provider) ?? listTtsProviders()[0];
return {
provider,
voice: prefs.ttsVoice ?? defaults.defaultVoice,
speed: prefs.ttsSpeed ?? 1,
model: prefs.ttsModel ?? defaults.defaultModel,
endpointPreset: prefs.ttsEndpointPreset ?? "openrouter",
endpoint: prefs.ttsEndpoint ?? "",
hasCredential: getTtsCredentialStatus(provider).hasCredential,
providers: listTtsProviders(),
};
}
export async function fetchTtsVoiceList(provider: TtsProviderId): Promise<TtsVoice[]> {
if (provider === "system" || provider === "piper") return [];
const credential = getTtsCredential(provider);
return fetchTtsVoices(provider, credential ?? undefined);
}
export async function speakTts(text: string, target?: "pet" | "prompt" | BrowserWindow): Promise<void> {
const prefs = getAppStateSnapshot().preferences;
const provider = prefs.ttsProvider ?? "system";
if (isInQuietHours()) {
debug("tts", "skipping speech because quiet hours are active");
return;
}
if (provider === "system") {
sendToTarget(target, { type: "tts-speak", text, voice: prefs.ttsVoice, rate: prefs.ttsSpeed });
return;
}
try {
const credential = getTtsCredential(provider);
const result = await synthesizeTts({
text,
provider,
voice: prefs.ttsVoice,
speed: prefs.ttsSpeed,
model: prefs.ttsModel,
endpointPreset: prefs.ttsEndpointPreset,
endpoint: prefs.ttsEndpoint,
});
if (result.audio && result.mimeType) {
sendToTarget(target, { type: "tts-audio", audio: result.audio, mimeType: result.mimeType });
} else {
// Provider is system or returned no audio; fall back to OS voice.
sendToTarget(target, { type: "tts-speak", text, voice: prefs.ttsVoice, rate: prefs.ttsSpeed });
}
} catch (error) {
logError("tts", "speech synthesis failed", { error: String(error) });
throw error;
}
}
export function stopTts(target?: "pet" | "prompt" | BrowserWindow): void {
sendToTarget(target, { type: "tts-stop" });
}
function resolveWindow(target: "pet" | "prompt" | BrowserWindow | undefined): BrowserWindow | null {
if (target instanceof Object && "webContents" in target) return target as BrowserWindow;
if (target === "prompt") return getPromptWindow();
return getDefaultPetWindowForPlugins();
}
function sendToTarget(target: "pet" | "prompt" | BrowserWindow | undefined, message: TtsRendererMessage): void {
const window = resolveWindow(target);
if (!window || window.isDestroyed()) return;
if (message.type === "tts-audio") {
window.webContents.send("openpets:tts-audio", { audio: message.audio, mimeType: message.mimeType });
} else if (message.type === "tts-speak") {
window.webContents.send("openpets:tts-speak", { text: message.text, voice: message.voice, rate: message.rate });
} else if (message.type === "tts-stop") {
window.webContents.send("openpets:tts-stop");
}
}
type TtsRendererMessage =
| { type: "tts-audio"; audio: Buffer; mimeType: string }
| { type: "tts-speak"; text: string; voice?: string; rate?: number }
| { type: "tts-stop" };

116
apps/desktop/src/windows.ts
View file

@ -3,7 +3,7 @@ import { join } from "node:path";
import { app, BrowserWindow, clipboard, dialog, ipcMain, protocol, shell, type IpcMainInvokeEvent, type OpenDialogOptions } from "electron";
import { getAgentSetupSnapshot, runAgentSetupAction, updateAgentSetupCommandPaths } from "./agent-setup.js";
import { buildOpenPetsMcpServerPreview, getAgentSetupSnapshot, runAgentSetupAction, testOpenPetsMcpServer, updateAgentSetupCommandPaths } from "./agent-setup.js";
import { refreshAgentPetContent } from "./agent-pet-controller.js";
import { getAppStateSnapshot, normalizeOpenApiChatEndpoint, normalizePetScale, petScaleOptions, updatePreferences } from "./app-state.js";
import { createAppIcon } from "./assets.js";
@ -14,6 +14,9 @@ import { recoverDefaultPetMouseInterop, refreshDefaultPetContent, resetDefaultPe
import { getMcpChatClientManager, listMcpChatVanillaServerIds } from "./mcp-chat-client.js";
import { installPersistentToolkit, type McpToolkitPersistentTarget } from "./mcp-toolkit-installer.js";
import { clearOpenApiCredential, getOpenApiChatSettingsSnapshot, normalizeChatModel, resetOpenApiConversationContext, saveOpenApiCredential } from "./openapi-chat.js";
import { clearTtsCredential, saveTtsCredential } from "./tts-credentials.js";
import { fetchTtsVoiceList, getTtsSettingsSnapshot, speakTts, stopTts, type TtsProviderId } from "./tts-service.js";
import { listTtsProviders } from "./tts-engine.js";
import { addKnowledgeMemory, deleteKnowledgeFile, listKnowledgeFiles, searchKnowledgeStore, storeKnowledgeFile } from "./knowledge-store.js";
import { forgetOpenPetsMemory, listOpenPetsMemories, searchOpenPetsMemories, storeOpenPetsMemory, updateOpenPetsMemory } from "./openpets-memory.js";
import { installPet, installPetFromFolder, installPetFromZipFile, removePet, setDefaultInstalledPet } from "./pet-installation.js";
@ -642,6 +645,56 @@ export function installInternalUiHandlers(): void {
assertAllowedSender(event, ["control-center"]);
return updateAgentSetupCommandPaths(patch);
});
ipcMain.handle("openpets:openpets-mcp-server-preview", (event, selectedPetId: unknown, commandMode: unknown) => {
assertAllowedSender(event, ["control-center"]);
const petId = typeof selectedPetId === "string" && selectedPetId.trim() ? selectedPetId.trim() : undefined;
const mode = typeof commandMode === "string" && (commandMode === "published" || commandMode === "bundled" || commandMode === "local") ? commandMode : "published";
return buildOpenPetsMcpServerPreview(petId, mode);
});
ipcMain.handle("openpets:test-openpets-mcp-server", async (event, selectedPetId: unknown, commandMode: unknown) => {
assertAllowedSender(event, ["control-center"]);
const petId = typeof selectedPetId === "string" && selectedPetId.trim() ? selectedPetId.trim() : undefined;
const mode = typeof commandMode === "string" && (commandMode === "published" || commandMode === "bundled" || commandMode === "local") ? commandMode : "published";
return testOpenPetsMcpServer(petId, mode);
});
ipcMain.handle("openpets:get-tts-settings", (event) => {
assertAllowedSender(event, ["control-center"]);
return getTtsSettingsSnapshot();
});
ipcMain.handle("openpets:get-tts-voices", async (event, provider: unknown) => {
assertAllowedSender(event, ["control-center"]);
if (typeof provider !== "string") throw new Error("Provider must be a string.");
return fetchTtsVoiceList(provider as TtsProviderId);
});
ipcMain.handle("openpets:save-tts-credential", (event, provider: unknown, credential: unknown) => {
assertAllowedSender(event, ["control-center"]);
if (typeof provider !== "string") throw new Error("Provider must be a string.");
if (typeof credential !== "string") throw new Error("Credential must be a string.");
return saveTtsCredential(provider, credential);
});
ipcMain.handle("openpets:clear-tts-credential", (event, provider: unknown) => {
assertAllowedSender(event, ["control-center"]);
if (typeof provider !== "string") throw new Error("Provider must be a string.");
return clearTtsCredential(provider);
});
ipcMain.handle("openpets:test-tts", async (event, text: unknown) => {
assertAllowedSender(event, ["control-center"]);
if (typeof text !== "string" || !text.trim()) throw new Error("Text must be a non-empty string.");
await speakTts(text.trim(), "pet");
return { ok: true };
});
ipcMain.handle("openpets:tts-stop", (event) => {
assertAllowedSender(event, ["control-center"]);
stopTts("pet");
});
}
async function chooseLocalPetImportKind(owner: BrowserWindow | undefined): Promise<"zip" | "folder" | null> {
@ -914,12 +967,12 @@ async function getDefaultPetPreviewSpriteInfo(): Promise<{ readonly path: string
return { path: builtInPath, version: `builtin-${Math.round(fallback.mtimeMs)}-${fallback.size}` };
}
function validatePreferencePatch(value: unknown): { openDefaultPetOnLaunch?: boolean; locale?: LocalePreference; petScale?: number; reactionAnimationOverrides?: ReturnType<typeof validateReactionAnimationOverrides>; openApiChatModel?: string | undefined; openApiChatSystemPrompt?: string | undefined; openApiChatEndpoint?: string | undefined; openApiChatTheme?: "system" | "light" | "dark" } {
function validatePreferencePatch(value: unknown): { openDefaultPetOnLaunch?: boolean; locale?: LocalePreference; petScale?: number; reactionAnimationOverrides?: ReturnType<typeof validateReactionAnimationOverrides>; openApiChatModel?: string | undefined; openApiChatSystemPrompt?: string | undefined; openApiChatEndpoint?: string | undefined; openApiChatTheme?: "system" | "light" | "dark"; ttsProvider?: TtsProviderId; ttsVoice?: string | undefined; ttsSpeed?: number; ttsModel?: string | undefined; ttsEndpointPreset?: "openrouter" | "litellm" | "wavespeedai" | "custom"; ttsEndpoint?: string | undefined } {
if (!isRecord(value)) {
throw new Error("Invalid preferences patch.");
}
const patch: { openDefaultPetOnLaunch?: boolean; locale?: LocalePreference; petScale?: number; reactionAnimationOverrides?: ReturnType<typeof validateReactionAnimationOverrides>; openApiChatModel?: string | undefined; openApiChatSystemPrompt?: string | undefined; openApiChatEndpoint?: string | undefined; openApiChatTheme?: "system" | "light" | "dark" } = {};
const patch: { openDefaultPetOnLaunch?: boolean; locale?: LocalePreference; petScale?: number; reactionAnimationOverrides?: ReturnType<typeof validateReactionAnimationOverrides>; openApiChatModel?: string | undefined; openApiChatSystemPrompt?: string | undefined; openApiChatEndpoint?: string | undefined; openApiChatTheme?: "system" | "light" | "dark"; ttsProvider?: TtsProviderId; ttsVoice?: string | undefined; ttsSpeed?: number; ttsModel?: string | undefined; ttsEndpointPreset?: "openrouter" | "litellm" | "wavespeedai" | "custom"; ttsEndpoint?: string | undefined } = {};
if ("openDefaultPetOnLaunch" in value) {
if (typeof value.openDefaultPetOnLaunch !== "boolean") throw new Error("Invalid open-on-launch value.");
@ -980,6 +1033,63 @@ function validatePreferencePatch(value: unknown): { openDefaultPetOnLaunch?: boo
patch.openApiChatTheme = value.openApiChatTheme;
}
if ("ttsProvider" in value) {
const validIds = listTtsProviders().map((p) => p.id);
if (!validIds.includes(value.ttsProvider as TtsProviderId)) {
throw new Error("Invalid TTS provider.");
}
patch.ttsProvider = value.ttsProvider as TtsProviderId;
}
if ("ttsVoice" in value) {
if (value.ttsVoice === undefined || value.ttsVoice === null || value.ttsVoice === "") {
patch.ttsVoice = undefined;
} else if (typeof value.ttsVoice === "string") {
patch.ttsVoice = value.ttsVoice;
} else {
throw new Error("Invalid TTS voice.");
}
}
if ("ttsSpeed" in value) {
const speed = Number(value.ttsSpeed);
if (!Number.isFinite(speed) || speed < 0.5 || speed > 2) {
throw new Error("Invalid TTS speed. Must be between 0.5 and 2.");
}
patch.ttsSpeed = Math.round(speed * 10) / 10;
}
if ("ttsModel" in value) {
if (value.ttsModel === undefined || value.ttsModel === null || value.ttsModel === "") {
patch.ttsModel = undefined;
} else if (typeof value.ttsModel === "string") {
const trimmed = value.ttsModel.trim();
if (trimmed.length > 120) throw new Error("Invalid TTS model name.");
patch.ttsModel = trimmed;
} else {
throw new Error("Invalid TTS model.");
}
}
if ("ttsEndpointPreset" in value) {
if (value.ttsEndpointPreset !== "openrouter" && value.ttsEndpointPreset !== "litellm" && value.ttsEndpointPreset !== "wavespeedai" && value.ttsEndpointPreset !== "custom") {
throw new Error("Invalid TTS endpoint preset.");
}
patch.ttsEndpointPreset = value.ttsEndpointPreset;
}
if ("ttsEndpoint" in value) {
if (value.ttsEndpoint === undefined || value.ttsEndpoint === null || value.ttsEndpoint === "") {
patch.ttsEndpoint = undefined;
} else {
const endpoint = normalizeOpenApiChatEndpoint(value.ttsEndpoint);
if (!endpoint) {
throw new Error("Invalid TTS endpoint. Use https, or http only for localhost, and provide a base ending in /v1 or a full URL.");
}
patch.ttsEndpoint = endpoint;
}
}
return patch;
}

32
apps/desktop/tests/tts-engine.test.ts Normal file
View file

@ -0,0 +1,32 @@
import assert from "node:assert/strict";
import { getOpenAiTtsVoices, getTtsEndpointPreset, getTtsProviderDefaults, listTtsProviders } from "../src/tts-catalog.js";
const providers = listTtsProviders();
assert.ok(providers.length > 0, "expected at least one TTS provider");
assert.ok(providers.some((provider) => provider.id === "system"), "expected system provider");
assert.ok(providers.some((provider) => provider.id === "openai"), "expected openai provider");
assert.ok(providers.some((provider) => provider.id === "openai-compatible"), "expected openai-compatible provider");
const systemDefaults = providers.find((provider) => provider.id === "system");
assert.equal(systemDefaults?.defaultModel, "");
assert.equal(systemDefaults?.defaultVoice, "");
assert.deepEqual(systemDefaults?.voices, []);
const openAiDefaults = providers.find((provider) => provider.id === "openai");
assert.equal(openAiDefaults?.defaultModel, "tts-1");
assert.equal(openAiDefaults?.defaultVoice, "alloy");
assert.ok(openAiDefaults?.voices.some((voice) => voice.id === "alloy"));
assert.deepEqual(getOpenAiTtsVoices().map((voice) => voice.id), ["alloy", "echo", "fable", "onyx", "nova", "shimmer"]);
assert.deepEqual(getTtsProviderDefaults("system"), { model: "", voice: "" });
assert.deepEqual(getTtsProviderDefaults("openai"), { model: "tts-1", voice: "alloy" });
assert.deepEqual(getTtsProviderDefaults("elevenlabs"), { model: "eleven_multilingual_v2", voice: "" });
assert.deepEqual(getTtsEndpointPreset("openrouter"), { endpoint: "https://openrouter.ai/api/v1", model: "" });
assert.deepEqual(getTtsEndpointPreset("litellm"), { endpoint: "http://localhost:4000/v1", model: "" });
assert.deepEqual(getTtsEndpointPreset("wavespeedai"), { endpoint: "https://api.wavespeed.ai/v1", model: "" });
assert.deepEqual(getTtsEndpointPreset("custom"), { endpoint: "", model: "" });
console.error("TTS engine validation passed.");

8
docs/pull-requests/OPENPETS_3_1_0_MOD_NICHE_PULL_REQUESTS.md
View file

@ -99,11 +99,12 @@ Target branch for all PRs: `v3.1.0-integrated` (or `main` once the OP 1–6 inte
## 4. MCP edition / Agent integrations niche
**Branch:** `pr/chat-memory-mcp-toolkit` (historical main PR)
**Scope:** MCP server, vanilla chat MCP tools, curated MCP Toolkit, and agent integrations.
**Scope:** OpenPets MCP server, MCP Tool Servers, OpenPets MCP Server panel, Curated MCP Toolkit, and agent integrations.
### What changed
- **OpenPets MCP server** (`packages/mcp/`) exposes `openpets_status`, `openpets_react`, `openpets_say`, and memory tools through the `@open-pets/mcp` package.
- **Vanilla Chat MCP tools** — internal stdio MCP client (`mcp-chat-client.ts`) with tiered activation UI and a ReAct-style tool loop.
- **MCP Tool Servers** — internal stdio MCP client (`mcp-chat-client.ts`) with tiered activation UI and a ReAct-style tool loop for the built-in floating chat.
- **OpenPets MCP Server panel** — central Control Center > Integrations card for command mode, node path, pet routing, server test, and MCP JSON copy.
- **Curated MCP Toolkit** — permission-aware reference panel in Control Center > Integrations with install orchestration for Claude/Codex baselines.
- **Agent integrations** — Claude Code, OpenCode, Cursor, and Pi setup flows; managed instructions/rules/MCP config previews.
@ -121,7 +122,8 @@ Target branch for all PRs: `v3.1.0-integrated` (or `main` once the OP 1–6 inte
- `08eff41` (main feature PR), `3f8fa1a` (MCP package baseline)
### Verification
- Open Integrations > MCP Toolkit and activate Filesystem + Fetch/Web.
- Open Integrations > MCP Tool Servers and activate Filesystem + Fetch/Web.
- Open Integrations > OpenPets MCP Server, choose a command mode, and click Test Server.
- In floating chat, ask "list my home directory files" — the activated tools should run.
- Right-click the pet → Virtual Pet ▸ Feed/Play/Pet/Nap; stats update and the context menu reflects the new values.