From 5c303b11288ce72e9d965d8bf69ceaf2f1e6dc29 Mon Sep 17 00:00:00 2001 From: Shashmit Kumar Date: Tue, 28 Jul 2026 23:20:30 +0530 Subject: [PATCH] docs: design local AI article studio --- ...26-07-28-local-ai-article-studio-design.md | 633 ++++++++++++++++++ 1 file changed, 633 insertions(+) create mode 100644 docs/superpowers/specs/2026-07-28-local-ai-article-studio-design.md diff --git a/docs/superpowers/specs/2026-07-28-local-ai-article-studio-design.md b/docs/superpowers/specs/2026-07-28-local-ai-article-studio-design.md new file mode 100644 index 00000000..bfc94c66 --- /dev/null +++ b/docs/superpowers/specs/2026-07-28-local-ai-article-studio-design.md @@ -0,0 +1,633 @@ +# Local AI Article Studio — Design + +Date: 2026-07-28 + +## Summary + +Build a personal, laptop-local web application that turns a topic queue and an +editorial knowledge library into researched article drafts. A deterministic +coordinator runs a team of specialized AI agents, saves every durable artifact +to MongoDB, and presents the resulting article and private research evidence for +human review. No article is published without explicit approval. + +The application, MongoDB, Supermemory, scheduling, and all article data run +locally. OpenRouter is the only required remote service. Gemini 3.5 Flash Lite +is the default model for both article agents and self-hosted Supermemory's +model-backed extraction steps. Supermemory uses local embeddings. + +Obsidian is not part of the system. The application owns the knowledge library, +editorial workflow, and approval experience. + +## Goals + +- Generate one or many articles from explicit instructions. +- Automatically prepare one daily draft from a manual topic queue, falling back + to an AI-proposed topic when the queue is empty. +- Keep company profiles, editorial rules, style guides, writing samples, and + approved articles in a searchable knowledge library. +- Use Supermemory to retrieve relevant style and company context and detect + semantically similar existing articles. +- Research current information on the web and retain evidence for private review. +- Run specialized AI roles with validated, inspectable inputs and outputs. +- Support editing, requested revisions, immutable version history, and explicit + approval in the web application. +- Recover safely from model errors, process restarts, and laptop sleep. +- Preserve a clean extension point for later website publishing. + +## Non-goals + +- Multi-user accounts, teams, billing, or tenant isolation. +- Cloud deployment or remote access. +- Automatic publishing in the first version. +- Obsidian import, synchronization, or export. +- Training or fine-tuning a model. +- Treating temporary web research as permanent knowledge automatically. +- Allowing autonomous agents to communicate or loop without coordinator limits. + +## Primary Design Decisions + +| Concern | Decision | +| --- | --- | +| User model | One local user | +| Application | New local Next.js application in the monorepo | +| Source of truth | MongoDB database `article_studio` | +| Semantic memory | Self-hosted Supermemory at `http://localhost:6767` | +| Embeddings | Supermemory's local embedding model | +| Remote AI | OpenRouter only | +| Default model | `google/gemini-3.5-flash-lite` | +| Orchestration | Deterministic workflow coordinator | +| Web research | OpenRouter web-search server tool behind an adapter | +| Publication | Manual approval only; publishing deferred | +| Visual documentation | Markdown diagrams and comparisons | + +## Repository Placement + +Article Studio remains isolated from the hosted Supermemory product: + +```text +apps/ +├── article-studio/ # Next.js UI and local API, port 3010 +└── article-studio-worker/ # Long-running Bun scheduler and workflow worker + +packages/ +└── article-studio-core/ # Domain, persistence, agents, orchestration, adapters +``` + +The existing `apps/web` is not reused because it is coupled to hosted +authentication, billing, analytics, and Cloudflare deployment. The new web +application uses the Node.js runtime for MongoDB access and does not inherit +OpenNext/Cloudflare configuration. + +The new workspaces declare compatible dependencies explicitly. They standardize +on AI SDK 6 and Zod 4 internally rather than depending on the monorepo's older +root versions or the hosted API validation package. The UI application reuses +the app-neutral primitives and theme tokens in `packages/ui`, `packages/lib`, +and `packages/hooks`. Existing app-specific editor code may be extracted into a +focused shared component, but the new app never imports source directly from +another application. + +## Local Configuration + +The Article Studio processes read local environment variables: + +```env +ARTICLE_STUDIO_PORT=3010 +ARTICLE_STUDIO_TIMEZONE=Asia/Kolkata +MONGODB_URI=mongodb://127.0.0.1:27017/?retryWrites=false +MONGODB_DATABASE=article_studio +SUPERMEMORY_BASE_URL=http://127.0.0.1:6767 +SUPERMEMORY_API_KEY= +OPENROUTER_API_KEY= +OPENROUTER_MODEL=google/gemini-3.5-flash-lite +``` + +Self-hosted Supermemory is configured to use the same OpenRouter account for its +model-backed extraction while retaining local embeddings: + +```env +OPENAI_BASE_URL=https://openrouter.ai/api/v1 +OPENAI_API_KEY= +OPENAI_MODEL=google/gemini-3.5-flash-lite +SUPERMEMORY_EMBEDDING_PROVIDER=local +``` + +Secrets stay in ignored local environment files or the operating-system +environment. MongoDB stores only redacted configuration status and non-secret +workflow snapshots. + +## System Architecture + +```text +┌────────────────────────── Laptop ──────────────────────────────┐ +│ │ +│ Browser │ +│ │ │ +│ ▼ │ +│ Local Next.js web app │ +│ ├── API layer │ +│ ├── knowledge and editorial UI │ +│ ├── deterministic workflow coordinator │ +│ └── local scheduler/worker │ +│ │ │ │ +│ ▼ ▼ │ +│ MongoDB 8.2.12 Supermemory local │ +│ article_studio http://localhost:6767 │ +│ exact records semantic retrieval + local embeddings │ +│ │ │ │ +└────────────┼────────────────────┼─────────────────────────────┘ + │ │ relevant context + └────────────────────┴───────────┐ + ▼ + OpenRouter API + Gemini 3.5 Flash Lite + + web research +``` + +The browser never connects directly to MongoDB, Supermemory, or OpenRouter. +Server-side modules own those connections and expose validated application APIs. + +## Component Boundaries + +### Web application + +Owns the browser experience, API routes, input validation, article editing, +review actions, and health displays. It never contains workflow logic inside UI +components. + +### Workflow coordinator + +Owns the state machine. It selects the next runnable step, creates an agent run, +validates the agent artifact, persists it, and advances the workflow. Agents do +not select other agents or mutate workflow state directly. + +### Worker and scheduler + +Runs generation outside request/response lifetimes. It polls for runnable work, +claims one workflow using an expiring lease, records a heartbeat, and executes +one step at a time. On startup it evaluates missed daily schedules and enqueues +at most one catch-up draft for each missed schedule window. + +Only one workflow runs concurrently by default. Each article in a multi-article +request has its own workflow, allowing independent retry and review. + +The worker is a separate long-running Bun application. It does not run inside +Next.js route handlers, development hot-reload processes, or browser requests. + +### MongoDB repositories + +Provide typed persistence interfaces. Application and agent modules do not issue +ad hoc MongoDB queries. Atomic single-document operations provide state changes, +leases, counters, and idempotency on the standalone MongoDB deployment. + +### Supermemory gateway + +Indexes selected knowledge and approved articles, retrieves relevant context, +and performs similarity searches. MongoDB remains the canonical store. Every +indexed item uses the MongoDB identifier as its stable Supermemory `customId`. +Because ingestion is asynchronous, the gateway records ingestion state and +polls for readiness before a workflow depends on newly changed knowledge. + +### Model gateway + +Uses the Vercel AI SDK and the OpenRouter provider for normal structured agent +generation. Model choice is configuration, not agent business logic. + +### Research provider + +Provides a narrow `search` capability that returns normalized source records. +The first implementation uses OpenRouter's `openrouter:web_search` server tool +with an explicit result cap. It calls OpenRouter's REST API directly because the +server tool is an OpenRouter-specific beta capability; normal agent generation +continues through Vercel AI SDK. The adapter boundary permits later replacement +without leaking provider-specific response annotations into domain code. + +### Publisher adapter + +Defines the future contract for publishing approved articles. Version one ships +with no external publisher and contains no automatic publish path. + +## Knowledge Model + +The application provides forms and editors for these knowledge types: + +- Company profile +- Style guide +- Editorial rule +- Writing sample +- Approved article +- Permanent reference + +Each knowledge item has a title, Markdown body, type, optional company +association, tags, status, content hash, MongoDB timestamps, and Supermemory +synchronization state. + +Supermemory indexes active company profiles, style guides, editorial rules, +writing samples, permanent references, and approved article versions. It does +not index rejected or unapproved drafts as positive examples. + +Temporary research remains scoped to its article because web facts can become +stale. A reviewer may explicitly promote a research item into the permanent +knowledge library. + +Before an article is outlined, the Memory Librarian retrieves: + +```text +global editorial rules ++ selected company profile ++ matching style guidance ++ relevant approved articles and writing samples ++ semantically similar topics/articles for duplication detection +``` + +Retrieved context is bounded and recorded with the workflow so a draft can be +audited without depending on later memory changes. + +## MongoDB Design + +Runtime configuration: + +```env +MONGODB_URI=mongodb://127.0.0.1:27017/?retryWrites=false +MONGODB_DATABASE=article_studio +``` + +MongoDB remains bound to localhost. The current no-authentication configuration +is accepted only while the application is local and single-user. The supplied +MongoDB 8.2.12 container and its persistent `mongodb_data` volume are used as +provided. Startup performs a version and capability check rather than assuming +transactions, change streams, or retryable writes are available. + +Documents use UUID strings consistently across MongoDB and Supermemory, BSON +dates, a `schema_version`, and explicit schema enums. Each workflow snapshots +its effective schedule, prompt, model, and research settings so later settings +changes cannot alter an in-flight run. + +### Collections + +| Collection | Responsibility | +| --- | --- | +| `knowledge_items` | Bounded knowledge content, canonical key, revision, and sync state | +| `topics` | Candidate topic, dedupe key, provenance, score, and lifecycle | +| `articles` | Identity plus mutable current/approved version pointers | +| `article_versions` | Immutable content snapshot, parent, stage, and content hash | +| `workflow_runs` | Authoritative state machine, settings snapshot, lease, fence, and retry | +| `agent_runs` | One structured agent attempt with hashes, usage, cost, and errors | +| `research_sources` | Normalized URL plus only the retrieved evidence excerpts used | +| `settings` | Versioned global schedule, generation, model, and feature settings | + +### Important indexes + +- `knowledge_items`: unique canonical key; status and updated time; tags. +- `topics`: partial unique dedupe key for proposed/queued/in-progress topics; + compound queue index on status, descending priority, and creation time. +- `articles`: unique topic identifier; unique slug; status and updated time; + company and status. +- `article_versions`: unique article and version number; workflow and created + time; content hash. +- `workflow_runs`: unique request/idempotency key; state, next-attempt time, and + creation time; lease expiry; unique partial index on a present `active_slot`. +- `agent_runs`: unique workflow, step, and attempt; workflow and start time. +- `research_sources`: unique workflow and source key; canonical URL and fetch + time. + +Large agent and research artifacts remain outside `articles` so version history +cannot grow one document without bound. Full fetched web pages are never stored. +If diagnostic transcripts approach MongoDB's document limit, they are written to +a bounded local artifact store and MongoDB retains only their hash and path. + +`workflow_runs` is the authoritative aggregate. Cross-collection records are +idempotent, repairable projections because the standalone database cannot +atomically update multiple documents. Article versions are insert-only by +application policy, carry a content hash, and are never updated in place. + +## Article and Workflow State + +Article states: + +```text +draft + → ready_for_review + ├── changes_requested → draft + └── approved +``` + +`archived` is available from any non-running editorial state. Future publishing +states will be added after `approved`; they are not simulated in version one. + +Workflow states: + +```text +queued + → running + ├── retry_wait → running + ├── needs_attention + ├── canceled + └── awaiting_approval + └── finalizing + └── completed +``` + +Only actively executing/finalizing work carries +`active_slot: "article-generation"`. A unique partial index enforces the +single-worker limit even across processes. Waiting, retrying, and +`awaiting_approval` workflows release the slot so one pending review cannot +block later drafts. + +Topic states are `proposed`, `queued`, `in_progress`, `used`, and `rejected`. +Only accepted `queued` topics can start a scheduled workflow. + +## Agent Team and Artifacts + +| Agent | Responsibility | Required artifact | +| --- | --- | --- | +| Topic Strategist | Select queued topic or propose a non-duplicate candidate | Topic proposal and rationale | +| Memory Librarian | Retrieve company, style, examples, and similarity evidence | Bounded context pack | +| Web Researcher | Find current, credible sources and claims | Research brief and source cards | +| Outline Architect | Design the article argument and section sequence | Structured outline | +| Article Writer | Write the complete initial or revised draft | Markdown article draft | +| Fact Checker | Map factual claims to evidence and flag unsupported claims | Claim-evidence report | +| Style and Brand Editor | Align voice, structure, terminology, and positioning | Revised draft and change log | +| SEO Reviewer | Suggest title, description, slug, keywords, and readability edits | SEO metadata and issue list | + +All artifacts are schema-validated with Zod. Agents receive only their required +inputs. They return artifacts to the coordinator and never call one another. + +Gemini 3.5 Flash Lite is the default for every role. Settings retain a per-role +model field so roles can be reassigned later without changing the coordinator. +Every agent run records its actual model, token usage, estimated cost, duration, +attempt number, and terminal status. + +## Generation Flows + +### Manual generation + +The user supplies a topic or instructions, company, article count, desired +length, and optional schedule date. The application creates one independent +topic, article shell, and workflow per requested article. If multiple articles +share a broad instruction, the Topic Strategist must produce distinct angles +before any research begins. + +### Daily generation + +```text +Daily scheduler + → claim highest-priority due queued topic + → if none exists, ask Topic Strategist for a proposal + → reject candidates too similar to existing work + → create article and workflow + → run editorial pipeline + → place article in review inbox +``` + +Automatic generation means automatic drafting only. Approval always requires a +user action in the review UI. + +### Editorial pipeline + +```text +Topic Strategist + → Memory Librarian + → Web Researcher + → Outline Architect + → Article Writer + → Fact Checker + → Style and Brand Editor + → SEO Reviewer + → ready_for_review +``` + +The Fact Checker can request at most two Writer revision cycles. A Style Editor +revision is fact-checked again if it changes factual claims. The coordinator +enforces all loop limits. + +### Approval + +Approval must match both `awaiting_approval` and the exact candidate version, +which prevents approval of a stale draft. The decision has its own idempotency +key. It moves the authoritative workflow to `finalizing`; the worker creates or +repairs the approved version pointer, enqueues idempotent Supermemory indexing, +and then marks the workflow complete. The final approved text remains in the +app. Research sources stay private and are excluded from exported article +content. + +## Web Research + +The Researcher uses the OpenRouter server-side web-search tool through the +`ResearchProvider` abstraction. Initial limits are: + +- Up to three search actions per article. +- Up to five results per search action. +- Up to fifteen total results. +- Normalized URL deduplication. +- Retrieval timestamps on every source. + +Each source card stores URL, title, publisher/domain, retrieved time, relevant +evidence excerpt, and the claims it supports. The research brief distinguishes +source-backed facts from model synthesis. + +Sources appear only in the review workspace. The final Markdown article contains +no automatic inline citations or source list. + +## User Interface + +Primary navigation: + +```text +Dashboard +├── Create articles +├── Topic queue +├── Review inbox +├── Article library +├── Knowledge library +└── Settings +``` + +### Dashboard + +Shows today's scheduled article, current workflow, failures requiring attention, +pending review count, recent approvals, and MongoDB/Supermemory/OpenRouter health. + +### Create articles + +Captures topic or instructions, company, number of articles, target length, and +optional scheduled date. A submission preview makes the number of workflows and +estimated generation scope explicit. + +### Topic queue + +Supports adding, reordering, pausing, accepting, and rejecting topics. Manual +queued topics take priority over AI proposals. + +### Review inbox + +Lists articles ready for review and highlights unsupported claims, duplication +warnings, and failed quality checks. + +### Review workspace + +```text +┌───────────────────────────────────────────────────────────────┐ +│ Article title Status: Ready for review │ +├─────────────────────────────────────┬─────────────────────────┤ +│ │ Sources │ +│ Editable Markdown article │ Agent reports │ +│ │ Similarity check │ +│ │ Version history │ +│ │ Cost and token usage │ +├─────────────────────────────────────┴─────────────────────────┤ +│ Request changes Regenerate section Approve article │ +└───────────────────────────────────────────────────────────────┘ +``` + +Requesting changes captures reviewer instructions and creates a new immutable +version after revision. Regenerating a section preserves the previous version. + +### Article library + +Supports filtering, searching, viewing version history, archiving, and exporting +approved content as Markdown or JSON. + +### Knowledge library + +Supports creating, editing, activating, and archiving company profiles, style +guides, rules, samples, and references. It shows Supermemory synchronization +status without exposing raw vector details. + +### Settings + +Contains schedule, generation defaults, model selection, research limits, +Supermemory connection health, MongoDB connection health, and a disabled future +publishing section. API secrets remain in local environment configuration and +are never returned to the browser or stored in article records. + +## Error Handling and Recovery + +- A workflow lease has an owner, incrementing epoch/fence, heartbeat, and expiry. +- Claiming work uses an atomic conditional update and database-side time. +- Expired leases become runnable after restart. +- Every workflow commit matches the workflow revision, lease owner, and lease + epoch so a late worker cannot attach output after losing its lease. +- Each logical operation and attempt has a deterministic idempotency key. +- Each step reserves its attempt in `workflow_runs`, writes its artifact + idempotently, and then attaches the artifact with a fenced update. +- Startup reconciliation completes missing projection updates and identifies + unattached artifacts after a crash. +- Transient external failures receive two retries with bounded backoff. +- Retry time and deterministic jitter are persisted; correctness never depends + on an in-memory timer. +- Invalid model output receives one schema-repair attempt. +- Exhausted retries move the workflow to `needs_attention`. +- A human can retry the failed step or cancel the workflow. +- The coordinator never repeats a completed step unless the user explicitly + requests regeneration. +- Daily catch-up scheduling cannot create duplicate drafts. +- Supermemory synchronization failures do not discard approved articles; they + remain visible as an actionable sync error. +- Unapproved content cannot reach the publisher adapter. +- Approval conditionally matches the current candidate version and records a + decision id, preventing stale or repeated approval. +- Remote model and search calls are at-least-once: a crash after the provider + responds but before persistence may repeat cost, but fencing prevents duplicate + workflow advancement. + +## Cost and Resource Controls + +- One concurrent workflow by default. +- Search and revision-loop caps are enforced server-side. +- Context packs have per-category and overall size limits. +- Agent token use and estimated cost are persisted and shown in the review UI. +- Manual multi-article requests display their count before enqueueing. +- The worker can be paused without losing queued work. +- Model assignment is configurable per role, while Gemini 3.5 Flash Lite remains + the initial default everywhere. + +## Security and Privacy + +- MongoDB, the web app, and Supermemory bind to localhost. +- Durable workflow and version writes use acknowledged journaled writes. +- Only the app server holds OpenRouter and Supermemory credentials. +- Secrets are loaded from local environment configuration and never logged. +- Source excerpts and relevant memory context are sent to OpenRouter during + generation; the application is local-first, not fully offline. +- HTML from research sources is treated as untrusted input and converted to + plain evidence text before model use or UI rendering. +- Any future non-local deployment requires authentication, MongoDB credentials, + CSRF protection, hardened secret storage, and an explicit security review. + +## Testing Strategy + +### Unit tests + +- Agent input and output schemas. +- Topic normalization and similarity thresholds. +- State transitions and approval guards. +- Retry, lease-expiry, and idempotency behavior. +- Context-pack limits and source normalization. +- Token and estimated-cost calculations. + +### Integration tests + +- MongoDB repositories against a disposable test database. +- Supermemory gateway with a mock server matching the local API. +- Model gateway with recorded or mocked OpenRouter responses. +- Research provider normalization and source annotations. +- Worker recovery after an interrupted step. + +### End-to-end tests + +- Manual single-article generation. +- Multi-article request producing distinct workflows. +- Daily queued-topic selection. +- AI topic fallback when the queue is empty. +- Failure, `needs_attention`, and retry. +- Request changes, section regeneration, and immutable versions. +- Approval and Supermemory indexing. +- Proof that unapproved content cannot invoke publishing. + +## Health and Operations + +The dashboard reports: + +- MongoDB ping and database name. +- Supermemory API reachability. +- OpenRouter configuration status without revealing the key. +- Worker heartbeat and paused/running state. +- Oldest queued workflow and any expired leases. + +The `mongodb_data` Docker volume provides persistence but is not a backup. +Version one documents a local `mongodump`/`mongorestore` procedure for recovery. +Startup also applies idempotent schema/index migrations and runs workflow +reconciliation before the worker claims new jobs. + +## Future Publishing Extension + +The future `PublisherAdapter` accepts only an approved immutable article version +plus explicit publishing metadata. It returns an external identifier, URL, and +publication timestamp. Adding a website publisher must not alter the generation +pipeline or allow approval bypass. + +Potential future article states are: + +```text +approved + → scheduled_for_publish + → publishing + ├── published + └── publish_failed +``` + +## Acceptance Criteria + +- The local app can create, edit, and search knowledge items. +- Active knowledge synchronizes idempotently to local Supermemory. +- A manual request can generate one or multiple independent article workflows. +- A daily scheduler can choose a queued topic or create a non-duplicate proposal. +- All eight agent roles create schema-valid, inspectable artifacts. +- Web research produces private source cards linked to factual claims. +- Failed or interrupted workflows resume without duplicating completed work. +- The review UI supports editing, requesting changes, version comparison, and + explicit approval. +- Approved articles are indexed into Supermemory and exportable as Markdown/JSON. +- Drafts and research sources remain private. +- No unapproved article can enter a publishing path. +- MongoDB, Supermemory, and worker health are visible locally.