commit befea97bf63ef216a7e9560710a63422a7c0bbda Author: Himanshu Dongre Date: Sun Mar 22 13:48:50 2026 +0530 Initial public release diff --git a/.env.example b/.env.example new file mode 100644 index 0000000..ec98c23 --- /dev/null +++ b/.env.example @@ -0,0 +1,12 @@ +# Smriti - Environment Variables +# Copy this file to .env and fill in the values + +# Database +DATABASE_URL=postgresql://smriti:smriti@localhost:5432/smriti + +# OpenAI (leave empty to use mock provider) +OPENAI_API_KEY= +OPENAI_MODEL=gpt-4o-mini + +# App +DEBUG=false diff --git a/.github/ISSUE_TEMPLATE/bug_report.md b/.github/ISSUE_TEMPLATE/bug_report.md new file mode 100644 index 0000000..dac753b --- /dev/null +++ b/.github/ISSUE_TEMPLATE/bug_report.md @@ -0,0 +1,32 @@ +--- +name: Bug report +about: Something is not working as expected +labels: bug +--- + +**What happened?** + +A clear, concise description of the bug. + +**Expected behavior** + +What you expected to happen. + +**Steps to reproduce** + +1. ... +2. ... +3. ... + +**Environment** + +- Backend version (git hash or tag): +- Browser: +- Provider in use (openai / anthropic / openrouter / mock): +- Docker or local setup? + +**Relevant logs or error output** + +``` +paste here +``` diff --git a/.github/ISSUE_TEMPLATE/feature_request.md b/.github/ISSUE_TEMPLATE/feature_request.md new file mode 100644 index 0000000..7bfa943 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/feature_request.md @@ -0,0 +1,22 @@ +--- +name: Feature request +about: Propose a change or addition to Smriti +labels: enhancement +--- + +**What problem does this solve?** + +A clear description of the problem or gap this addresses. + +**Proposed approach** + +How you would implement it, or how you would like it to behave. + +**Alternatives considered** + +Other approaches you thought of and why you ruled them out. + +**Scope notes** + +Is this additive, or does it change existing behavior? Does it interact with +checkpoint isolation, the provider abstraction, or the lineage graph? diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md new file mode 100644 index 0000000..912d279 --- /dev/null +++ b/.github/pull_request_template.md @@ -0,0 +1,19 @@ +## Summary + +What does this PR do? + +## Changes + +- +- + +## Test coverage + +How was this tested? (backend tests, manual QA, typecheck, build) + +## Checklist + +- [ ] `make test` passes +- [ ] `npx tsc --noEmit` is clean (frontend changes) +- [ ] No secrets or credentials added +- [ ] Docs updated if behavior changed diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..29eddf8 --- /dev/null +++ b/.gitignore @@ -0,0 +1,51 @@ +# Python +__pycache__/ +*.py[cod] +*.egg-info/ +*.egg +dist/ +build/ +.venv/ +venv/ +*.so + +# Environment +.env + +# Node +node_modules/ +npm-debug.log* + +# IDE +.vscode/ +.idea/ +*.swp +*.swo +*~ + +# OS +.DS_Store +Thumbs.db + +# Docker +docker-compose.override.yml + +# Pytest +.pytest_cache/ +.coverage +htmlcov/ + +# Alembic +# Keep versions dir but not .pyc inside it +backend/config/providers.yaml + +# Logs — never stage +runtime_logs/ +test_artifacts/ +backend/backend_logs.txt +backend/backend.log +test_diagnostics.log +*.log + + + diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md new file mode 100644 index 0000000..1a9ae9d --- /dev/null +++ b/ARCHITECTURE.md @@ -0,0 +1,353 @@ +# Smriti — Architecture + +This document describes the system model, the checkpoint isolation mechanism, the +provider abstraction, and the API versioning strategy. It is a reference for +contributors and for anyone who wants to understand why Smriti behaves the way it does. + +--- + +## The Foundational Distinction: Event Stream vs. State Snapshot + +Every AI conversation produces two fundamentally different kinds of data: + +**Event stream** — the sequence of Turns. User sends a message; assistant replies. +Each Turn has a role, content, provider, model, and sequence number. The event stream +is append-only and never modified. It is the historical record of what was said. + +**State snapshot (Checkpoint)** — a structured, immutable summary of what was +concluded at a specific point. It contains: title, objective, summary, decisions, +tasks, open questions, entities. Once saved, a Checkpoint never changes. + +Most AI tools expose only the event stream. Smriti makes the state snapshot a +first-class, versioned object. This distinction is the architectural foundation of +everything else. + +``` +EVENT STREAM (turns) STATE SNAPSHOTS (checkpoints) +───────────────────── ────────────────────────────── +Turn 1 user Checkpoint A ── title +Turn 2 assistant (saved at objective +Turn 3 user turn 8) summary +Turn 4 assistant decisions[] +Turn 5 user tasks[] +Turn 6 assistant open_questions[] +Turn 7 user entities[] +Turn 8 assistant + ↑ checkpoint created here Checkpoint B ── title +Turn 9 user (saved at objective +Turn 10 assistant turn 14) summary +Turn 11 user decisions[] +... ... +``` + +A Checkpoint is not a transcript slice. It is not "turns 1–8 in summarized form." +It is a structured extraction of what was explicitly decided and understood — +extracted by the user (with optional AI drafting assistance), reviewed, and committed. +The extraction process has no access to anything outside the active conversation window. + +--- + +## Data Model + +### Space (RepoModel) + +The long-lived container. Fields: `id`, `name`, `description`, `user_id`, +`created_at`, `updated_at`. + +A Space holds many Checkpoints and many Sessions. It does not hold conversation +content directly; it is the namespace. + +### Session (ChatSession) + +The live chat runtime. Fields: `id`, `repo_id` (nullable), `title`, +`active_provider`, `active_model`, `seeded_commit_id`, `created_at`, `updated_at`. + +`repo_id` — null if the Session is not attached to a Space (FRESH mode). + +`seeded_commit_id` — the Checkpoint this Session was initialized from, if any. +Currently informational; not used as a hard isolation boundary by itself. + +A Session is not bound to a provider. The `active_provider` and `active_model` fields +reflect the last provider used, but any Turn can be created with any configured +provider. + +### Turn (TurnEvent) + +The event stream record. Fields: `id`, `session_id`, `repo_id`, `role`, `content`, +`provider`, `model`, `sequence_number`, `created_at`. + +`sequence_number` is assigned at write time, incrementing within the Session. It is +the key field used for checkpoint isolation (see below). + +`role` is `"user"` or `"assistant"`. System-role Turns are filtered out of context +construction. + +Turns are never deleted or updated. + +### Checkpoint (CommitModel) + +The state snapshot. Fields: `id`, `repo_id`, `commit_hash`, `parent_commit_id`, +`branch_name`, `message`, `objective`, `summary`, `decisions[]`, `tasks[]`, +`open_questions[]`, `entities[]`, `author_agent`, `metadata_`, `created_at`. + +`commit_hash` — a SHA-256 derived from repo ID, message, and creation timestamp. +Stable identifier for display (first 7 characters shown in the UI). + +`parent_commit_id` — links to the previous Checkpoint in the same Space. Forms an +implicit chain used for ancestor walking in `latest_3` scope mode. + +`metadata_` — JSONB field; currently stores `session_id` of the Session that +created the Checkpoint. Reserved for future use (source Turn range, etc.). + +--- + +## The Three Context Modes + +When a message is sent, `send_message` determines what context to inject into the +prompt based on three variables: + +1. Whether a Space is attached (`repo_id`) +2. Whether a specific Checkpoint is mounted (`mounted_checkpoint_id`) +3. The memory scope (`memory_scope`: `latest_1` or `latest_3`) + +### FRESH + +`repo_id` is null. No Checkpoint context. The prompt contains only the current user +message. The model has no prior structured state. + +### HEAD + +`repo_id` is set, `mounted_checkpoint_id` is null. + +`_resolve_checkpoints()` fetches the N most recent Checkpoints from the Space +(`latest_1` → 1, `latest_3` → 3), ordered oldest-first. + +Turn history filter: `TurnEvent.created_at >= latest_checkpoint.created_at`. +This passes Turns created since the most recent Checkpoint was saved — the +"work since last checkpoint" window. + +### MOUNTED + +`mounted_checkpoint_id` is set to a specific Checkpoint ID. + +`_resolve_checkpoints()` uses `_walk_ancestors()` to build the chain: the mounted +Checkpoint plus up to N-1 ancestors via `parent_commit_id`, oldest-first. + +Turn history filter: `TurnEvent.sequence_number > history_base_seq`. + +`history_base_seq` is provided by the frontend and represents the `sequence_number` +of the last Turn that existed at the moment the user clicked Mount. Only Turns +created after that moment are included. This is the isolation mechanism. + +--- + +## Checkpoint Isolation — The Mechanism + +This is the most important behavioral guarantee in Smriti. + +**The problem it solves:** + +A user works in Session A, creates Checkpoint 1 (Australia trip), continues working, +creates Checkpoint 2 (Australia + New Zealand). Later, the user mounts Checkpoint 1 +and asks a question. Without isolation, the Turn history filter `created_at >= +checkpoint1.created_at` would include all the New Zealand turns, because they were +created after Checkpoint 1 was saved. The model would see NZ context it should not. + +**The solution:** + +When the user clicks Mount in the UI, the frontend records: + +```typescript +const mountedAtSeq = Math.max(...turns.map(t => t.sequence_number ?? 0)); +``` + +This is the sequence number of the last Turn that existed before mounting. It is +passed as `history_base_seq` on every subsequent `send_message` call. + +The backend applies: + +```python +if payload.mounted_checkpoint_id is not None and payload.history_base_seq is not None: + history_stmt = history_stmt.where( + TurnEvent.sequence_number > payload.history_base_seq + ) +``` + +Effect: Turns 1–12 (pre-mount) are excluded. Only Turns 13+ (typed after mounting) +are included. The NZ conversation is turn 5–12 and does not enter the context. + +**Why sequence number, not timestamp?** + +Timestamps can have sub-second collisions and are harder to reason about precisely. +Sequence numbers are assigned by the backend at write time, monotonically increasing +within a session. `sequence_number > N` is an exact, collision-free boundary. + +**Current limitation:** + +The isolation is enforced within a single Session. Mounting Checkpoint 1 in Session A +uses Turns from Session A only (filtered by sequence boundary). This is correct +behavior. A future improvement (v5.3) will support true session forking: mounting a +Checkpoint creates a new Session with a clean Turn history, making the isolation +architecturally complete rather than boundary-based. + +**The same isolation applies to Draft with AI:** + +`draft_checkpoint` in `checkpoint.py` uses the same `_fetch_turns_for_draft()` +function that applies the identical `sequence_number > history_base_seq` filter. +A Checkpoint draft created while in MOUNTED mode sees only the same Turn window +as the chat session does. The two are consistent. + +--- + +## Context Construction + +When `send_message` is called, the prompt is built by `build_prompt_from_checkpoints()`: + +``` +You are continuing a conversation. + +Checkpoint [N] Summary: + + +Checkpoint [N] Key Decisions: +- +... + +Checkpoint [N] Open Tasks: +- +... + +Recent Conversation: +User: +Assistant: +... + +User Query: + +``` + +The prompt does not pass raw Turn content for Turns before the isolation boundary. +It passes structured Checkpoint fields (summary, decisions, tasks) for context, then +only the post-boundary Turn history as conversational context. + +This is a deliberate design: the model receives structured facts, not a raw log dump. +Noise in the event stream does not reach the model — only what was explicitly +crystallized into a Checkpoint. + +--- + +## Draft with AI — Extraction Quality + +The `draft_checkpoint` endpoint (`POST /api/v5/checkpoint/draft`) calls the +background intelligence model with a single extraction prompt. Key design constraints: + +**No prior context injection.** The prompt contains only the current conversation +transcript. No previous Checkpoint's decisions, tasks, or summary are injected. This +prevents cross-session contamination — a real failure mode where old product decisions +(e.g. "Use Postgres") appeared in travel planning checkpoints because they were in the +prior HEAD Checkpoint's fields. + +**Extract only, do not infer.** The prompt explicitly instructs: "Extract ONLY what is +discussed or decided in this conversation. Do NOT infer, hallucinate, or carry over +content from any other context. If a field has nothing relevant, return an empty array." + +**Decisions field semantics.** The prompt defines decisions as "explicit choices made +in the conversation, not hypothetical ones." This reduces false positives. + +**Replace, not merge.** Each Draft with AI call replaces all form fields. There is +no accumulation of previous drafts. The user sees a fresh extraction on every call. + +--- + +## Provider Abstraction + +All LLM calls go through a common adapter interface defined in +`backend/app/providers/registry.py`. Each provider (OpenAI, Anthropic, OpenRouter) +implements `send(messages, model, **kwargs) -> str`. + +OpenRouter uses the OpenAI SDK with a custom `base_url`. Anthropic uses the Anthropic +SDK. The adapter normalizes the message format and error handling. + +The abstraction has two important properties: + +1. **The session does not depend on the provider.** Turns are stored by Smriti, not + by the provider. Switching from OpenAI to Anthropic mid-session loses nothing. + The next request reconstructs the context prompt from Smriti's Turn and Checkpoint + records, the same way it would for any provider. + +2. **Background intelligence is a separate slot.** The model used for Draft with AI + and session auto-titling is configured independently in `background_intelligence` + in `providers.yaml`. It does not have to be the same provider the user is chatting + with. This allows a cheap, fast model (e.g. `gpt-4o-mini`) for extraction tasks + while the user interacts with a more capable model for reasoning. + +--- + +## API Versioning + +The backend API has multiple version prefixes. Each represents a product-generation +pivot, not an incremental change. + +| Prefix | Status | Purpose | +|---|---|---| +| `/api/v1` | Legacy | Transcript paste ingestion → session/artifact pipeline. Not part of current workflow. | +| `/api/v2` | Legacy | Agent-push model: repos, commits, context packs. Direct API-to-API handoff. Not part of current UI workflow. | +| `/api/v4` | Current | Chat sessions, message sending, provider management. The primary API. | +| `/api/v5` | Current | Checkpoint drafting. Isolated from chat API by design. | + +V1 and V2 endpoints remain registered for compatibility. They are not used by the +current frontend. New development targets V4 and V5 exclusively. + +The split between V4 and V5 is intentional: checkpoint operations (which involve a +background LLM call and structured extraction) are separated from the real-time chat +path. This allows different latency budgets and error handling strategies for each. + +--- + +## Frontend State Management + +The frontend manages three pieces of state relevant to checkpoint isolation: + +```typescript +const [mountedCheckpointId, setMountedCheckpointId] = useState(null); +const [mountedAtSeq, setMountedAtSeq] = useState(null); +const [memoryScope, setMemoryScope] = useState('latest_1'); +``` + +`mountedCheckpointId` — the ID of the currently mounted Checkpoint, or null. + +`mountedAtSeq` — the sequence number of the last Turn at the moment of mounting. +Set when Mount is clicked; cleared when Unmount is clicked or the Space changes. + +`memoryScope` — `'latest_1'` or `'latest_3'`. Selected in the Attach Space modal. +Controls ancestor walking depth in `_resolve_checkpoints()`. + +Every `sendChatMessage` call passes all three values. The backend uses them to +determine which context mode applies and which Turns to include. + +The UI displays the active context mode in two places: +- The thread header badge: `FRESH` / `HEAD · ` / `MOUNTED · ` +- The composer status line: `ctx: (description)` + +These are always in sync with the actual context being sent to the backend. + +--- + +## What Is Not Yet In the Architecture + +**Source Turn range on Checkpoints.** There is no record of which Turn range produced +a given Checkpoint. The `metadata_` JSONB field on `CommitModel` is the intended +storage location. This would allow "show me the conversation that produced this +Checkpoint" — a useful debugging and audit feature. + +**Streaming.** All provider calls are synchronous request/response. The adapter +interface does not yet support streaming. Each Turn waits for the full response. + +**Checkpoint merging.** The lineage graph can represent divergent branches but there +is no operation to merge two Checkpoint lines back together. Merging structured fields +(decisions, tasks) is mechanically possible; the semantics of merging reasoning intent +are not yet defined. + +**Authentication and multi-user.** All Spaces currently belong to a single demo user +(`DEMO_USER_ID`). There is no authentication layer, no user registration, and no +per-user isolation. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 0000000..f3739b0 --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,115 @@ +# Contributing to Smriti + +Thanks for your interest in contributing. This document covers how to get the dev +environment running, how to run tests, and how to propose changes. + +--- + +## Dev environment + +**Prerequisites:** Python 3.11+, Node 18+, PostgreSQL 14+ + +```bash +git clone https://github.com/your-org/smriti +cd smriti + +# Copy environment config +cp .env.example .env + +# Copy provider config template +cp backend/config/providers.example.yaml backend/config/providers.yaml +# Edit providers.yaml to add at least one provider API key, +# or leave all keys empty and use Mock Mode in the UI. + +# Install backend deps + run migrations +make setup + +# Start backend (terminal 1) +make dev + +# Start frontend (terminal 2) +make dev-frontend +``` + +Open `http://localhost:5173`. + +--- + +## Running tests + +```bash +# All backend tests (uses mock provider — no API key required) +make test + +# TypeScript typecheck +cd frontend && npx tsc --noEmit + +# Frontend production build +cd frontend && npm run build +``` + +The integration test suite uses the deterministic mock adapter and an in-memory +SQLite database (via `conftest.py`). No live API keys are needed. + +--- + +## Code style + +**Backend:** Python code is formatted and linted with `ruff`. + +```bash +make lint # Check +make format # Fix in place +``` + +**Frontend:** TypeScript with strict type checking. ESLint is configured but not +enforced in CI yet. + +There are no hard style rules beyond what ruff and tsc enforce. Prefer clarity over +brevity. Keep functions short. Do not add comments that restate the code — only +comment non-obvious decisions. + +--- + +## Making changes + +1. Fork the repo and create a feature branch. +2. Make your changes. Keep commits focused: one logical change per commit. +3. Run backend tests and frontend typecheck before opening a PR. +4. Open a PR against `main`. Describe what changed and why. + +For larger changes (new feature, API change, refactor), open an issue first to discuss +scope before writing code. + +--- + +## Reporting bugs + +Open a GitHub issue. Include: + +- What you expected to happen +- What actually happened +- Steps to reproduce (minimal, if possible) +- Relevant log output (`make logs` for Docker, or terminal output) +- Backend version (git hash or tag) + +--- + +## Contributing demos and docs + +Docs live in `docs/` and at the repo root (`README.md`, `ARCHITECTURE.md`, +`DECISIONS.md`). Demo scenarios live in `demos/`. + +For doc fixes: open a PR directly. For new demo scenarios: follow the structure in +`demos/branching-reasoning-demo/` — a README, RUNBOOK, DEMO_SCRIPT, and +EXPECTED_OUTCOMES at minimum. + +--- + +## What we are not looking for + +- Automatic checkpointing logic — Checkpoints are intentionally manual. +- Authentication / multi-user support — deferred by design. +- Changes to legacy V1/V2 endpoints — these are retained for compatibility only. + +If you want to work on something that falls outside normal scope, open an issue first. diff --git a/DECISIONS.md b/DECISIONS.md new file mode 100644 index 0000000..26efc4d --- /dev/null +++ b/DECISIONS.md @@ -0,0 +1,163 @@ +# Smriti — Key Decisions + +This document records significant architectural and product decisions and the +reasoning behind them. It is not a changelog; it covers choices that might otherwise +be revisited or questioned without this context. + +--- + +## Product decisions + +### Why version-control semantics, not a memory database + +Early versions of Smriti were built around extracting "memories" from transcripts and +storing them in a retrievable database. This proved wrong for several reasons: + +- Memories extracted from text are ambiguous and often incorrect +- Retrieval-based systems require the user to trust the retrieval quality — an + invisible black box +- There is no notion of "state at a point in time" — you cannot ask what you knew + at a specific moment and get a reliable answer + +The version-control model is fundamentally different: it requires the user to +explicitly create a Checkpoint at a meaningful point. The state is user-defined, not +inferred. The history is deterministic and inspectable. You can return to any +Checkpoint and know exactly what context the model will receive. + +The tradeoff is deliberate friction: Checkpoints are not automatic. This is correct. +Automatic snapshotting produces noise. Signal requires judgment. + +### Why separation of chat and checkpoint as distinct concerns + +The V4 API handles sessions and message sending. The V5 API handles checkpoint +drafting. These are separate for good reason: checkpoint drafting involves a +background LLM call with different latency, error handling, and reliability +requirements than real-time chat. Keeping them in separate route modules and API +version prefixes makes this boundary explicit. + +### Why Spaces are user-facing but Repos internally + +The codebase uses `RepoModel`, `repo_id`, and `/repos` in the database and API. +The UI and all user-facing text uses "Space." This is not a naming inconsistency — +it is a deliberate translation. "Repo" is accurate internally (it is Git-inspired) +but confusing to users who may think it refers to a code repository. "Space" is +neutral and descriptive: a named container for a line of work. + +When reading the code: `repo` = `Space` in user-facing terms. + +### Why checkpoints are called "Checkpoints" not "Commits" + +"Commit" is accurate in the Git analogy and is used throughout the codebase +(`CommitModel`, `commit_hash`, `parent_commit_id`). In user-facing text, "Checkpoint" +is preferred because: + +- It implies a deliberate save point, not a routine code commit +- It does not suggest branching/merging is available (it is not yet) +- It maps more intuitively to the actual usage: "I'm at a good stopping point, let + me save my state" + +### Why the name Smriti + +Smriti (स्मृति) means memory in Sanskrit. It is short, meaningful, and directly +describes the product's intent. The name was chosen before the product's mental model +shifted from "memory storage" to "versioned state" — but remains appropriate because +the core promise is still about preserving and recovering structured knowledge. + +--- + +## Architectural decisions + +### Why checkpoint isolation uses sequence numbers, not timestamps + +The isolation boundary — which Turns to include when a specific Checkpoint is mounted +— is enforced by `sequence_number > history_base_seq`, not by a timestamp comparison. + +Timestamps can have sub-millisecond collisions. Sequence numbers are assigned +monotonically within a Session at write time. `sequence_number > N` is an exact, +collision-free predicate. This matters because the boundary must be reliable: a +single Turn that should have been excluded contaminating the context is a trust failure. + +### Why history_base_seq is frontend-owned + +`history_base_seq` is computed by the frontend (the max `sequence_number` of existing +Turns at mount time) and passed on every `send_message` request. It is not stored in +the database on the Session or Checkpoint. + +Alternative considered: store the boundary on the Session when mounting. This was +rejected because: +- Mounting is a UI concept, not a persistent session state change (in v5.2) +- The same Session can be unmounted and remounted to different Checkpoints without + a database write +- The frontend always has the Turn list and can compute the boundary locally + +The tradeoff: if the frontend sends an incorrect `history_base_seq`, the isolation +is wrong. This is acceptable for v5.2; v5.3 will address this by forking a new Session +on mount, making the isolation structural. + +### Why Draft with AI does not inject prior checkpoint context + +Earlier versions of the checkpoint draft prompt injected the HEAD Checkpoint's +decisions and tasks as "prior context" for the extraction. This caused contamination: +decisions from a previous, unrelated line of work would appear in the draft for a +new topic (e.g. "Use Postgres" appearing in a travel planning checkpoint). + +The current design: the extraction prompt contains only the current conversation +Turns. The LLM is instructed to extract only what is explicitly present. No prior +state is injected. This produces accurate extractions at the cost of losing +"continuity awareness" in the extraction — but that continuity is already captured +in the Checkpoint fields that the user saves and can reference. + +### Why provider switching does not create a new session + +When a user switches from OpenAI to Anthropic mid-session, Smriti does not fork the +Session. The same `session_id` continues. The new provider receives the same +Checkpoint context and Turn history as the previous provider would have. + +This is correct behavior. The provider is a rendering engine, not a state owner. +Smriti owns the state. The fact that Claude and GPT-4o have different internal session +concepts is irrelevant; their sessions are not used. Smriti reconstructs the context +from its own records on every `send_message` call. + +### Why transcript ingestion is a legacy feature + +V1 of Smriti was built around pasting a transcript from one tool and generating a +"context pack" to paste into another. This workflow was the initial wedge but proved +limited: + +- Paste-based ingestion is friction-heavy +- Transcript quality is highly variable +- The extracted "context pack" was text, not structured state +- There was no versioning, no isolation, no "go back to this point" + +The V4+ model replaced this: Smriti is the workspace, not a bridge between workspaces. +Users work inside Smriti directly, and the structured state lives there. + +The V1 endpoints remain registered and functional for backfill use cases. They are +not part of the current UI and are not under active development. + +### Why there is no automatic checkpointing + +The system could detect decision keywords, count turns, or use a background model +to decide when to auto-create a Checkpoint. This was considered and rejected: + +- Automatic state snapshots have unclear semantics — the user did not define them +- They produce noise in the Checkpoint history, making the history less trustworthy +- The value of a Checkpoint comes from it representing a human judgment about + significance, not a system heuristic + +The current recommendation signal (a subtle "consider checkpointing" indicator after +a significant turn count or keyword detection) is present as a nudge, not an action. + +--- + +## Open questions and deferred decisions + +- **Multi-user Spaces** — deferred. No auth, no user model. +- **Checkpoint merging** — combining state from two divergent Checkpoint lines is + not yet defined. The semantics are unclear: merging structured fields (decisions, + tasks) is mechanical; merging reasoning intent is not. +- **Source Turn range on Checkpoints** — there is no record of which Turn range + produced a given Checkpoint. The `metadata_` JSONB field is the intended storage + location once the use case is validated. +- **Streaming** — the adapter interface does not yet support streaming. All provider + calls are synchronous request/response. diff --git a/LICENSE b/LICENSE new file mode 100644 index 0000000..d5f2c2e --- /dev/null +++ b/LICENSE @@ -0,0 +1,21 @@ +MIT License + +Copyright (c) 2026 Smriti Contributors + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/Makefile b/Makefile new file mode 100644 index 0000000..0f9aa3e --- /dev/null +++ b/Makefile @@ -0,0 +1,77 @@ +.PHONY: help dev up down test test-unit test-int lint format migrate + +VENV := cd backend && source .venv/bin/activate && + +help: ## Show this help + @grep -E '^[a-zA-Z_-]+:.*?## .*$$' $(MAKEFILE_LIST) | sort | awk 'BEGIN {FS = ":.*?## "}; {printf "\033[36m%-20s\033[0m %s\n", $$1, $$2}' + +# ── Docker ─────────────────────────────────────────────────────────────────── + +up: ## Start all services + docker compose up -d + +up-build: ## Rebuild and start all services + docker compose up -d --build + +down: ## Stop all services + docker compose down + +logs: ## Follow logs + docker compose logs -f + +# ── Backend ────────────────────────────────────────────────────────────────── + +dev: ## Run backend dev server locally (requires venv) + $(VENV) uvicorn app.main:app --reload --port 8000 + +test: ## Run all backend tests + $(VENV) python -m pytest -v + +test-unit: ## Run backend unit tests + $(VENV) python -m pytest tests/unit/ -v + +test-int: ## Run backend integration tests + $(VENV) python -m pytest tests/integration/ -v + +test-cov: ## Run tests with coverage + $(VENV) python -m pytest --cov=app --cov-report=term-missing -v + +lint: ## Lint backend code + $(VENV) ruff check . + +format: ## Format backend code + $(VENV) ruff format . + +# ── Database ───────────────────────────────────────────────────────────────── + +migrate: ## Run database migrations + $(VENV) alembic upgrade head + +migration: ## Create a new migration (usage: make migration msg="add users table") + $(VENV) alembic revision --autogenerate -m "$(msg)" + +# ── Frontend ───────────────────────────────────────────────────────────────── + +dev-frontend: ## Run frontend dev server locally + cd frontend && npm run dev + +test-frontend: ## Run frontend tests + cd frontend && npm test + +build-frontend: ## Build frontend for production + cd frontend && npm run build + +install-frontend: ## Install frontend dependencies + cd frontend && npm install + +# ── Setup ──────────────────────────────────────────────────────────────────── + +install: ## Install all dependencies (creates venv) + cd backend && python3 -m venv .venv && source .venv/bin/activate && pip install -e ".[dev]" + cd frontend && npm install + +setup: ## Full local setup (venv + deps + migrations) + cp -n .env.example .env || true + $(MAKE) install + $(MAKE) migrate + @echo "\n✅ Setup complete! Run: make dev (backend) / make dev-frontend (frontend)" diff --git a/README.md b/README.md new file mode 100644 index 0000000..378edf9 --- /dev/null +++ b/README.md @@ -0,0 +1,354 @@ +# Smriti — Version control for reasoning. + +Conversations become structured state, not disposable logs. + +Smriti (स्मृति, *memory* in Sanskrit) is a versioned AI workspace. It separates the +ephemeral event stream of a conversation from the durable structured state that +conversation produces — and makes that state explicitly versioned, inspectable, and +portable across models. + +**In Smriti, the system owns the state. Models are interchangeable processors.** + +--- + +## The Problem + +Every current LLM interface — ChatGPT, Claude.ai, Perplexity — treats conversation +state as disposable. When you close a tab, switch a model, or start a new session, +your context evaporates. You can export a transcript, but a transcript is +undifferentiated text: no structure, no version history, no way to return to a prior +state cleanly. + +The practical consequences: + +- Switching models means re-explaining everything from scratch +- There is no way to ask "what did I know at this point?" +- Context is implicit and fragile — the model infers it, often incorrectly +- There is no version history of reasoning, only a log of messages +- Long conversations accumulate noise faster than signal + +These are not UI problems. They are structural: existing tools do not distinguish +between *what was said* and *what was concluded*. + +--- + +## What Smriti Does Differently + +Smriti introduces a hard separation between two layers: + +**The event stream** — Turns of conversation, ordered, append-only, ephemeral. + +**The state snapshot (Checkpoint)** — an immutable, structured summary of what was +decided and understood at a specific point: title, objective, summary, decisions, tasks, +open questions, entities. + +A Checkpoint is not a transcript. It is the extracted state of a thinking process at a +moment in time. Once created, it does not change. + +This separation enables: + +- **Returning to a prior state** — mount any Checkpoint; only Turns created after + mounting are included in context. Later work does not leak backward. +- **Cross-model continuity** — a Checkpoint created in a GPT-4o session can seed a + Claude or Gemini session. The state travels; the model does not matter. +- **Inspectability** — "what did I decide at this point?" has a concrete, queryable + answer. +- **Branching** — fork any Checkpoint into a new session and explore a diverging line + of reasoning without affecting the original thread. +- **Comparison** — diff two Checkpoints across any branches to see exactly where + reasoning diverged. + +Smriti does not eliminate ambiguity or hallucination. It makes reasoning inspectable, +versioned, and recoverable. + +--- + +## Core Concepts + +### Space + +A Space is the long-lived container for a line of work. It holds the full history of +Checkpoints created within that work and all Sessions associated with it. + +Analogy: a Git repository, but for a thinking process rather than a codebase. + +Spaces are named and persistent. You might have one for a product architecture +decision, another for a research topic, another for a client engagement. + +### Session + +A Session is a live chat runtime. It may be attached to a Space (and thus have access +to that Space's Checkpoint history) or standalone (no persistent state). + +A Session has an active provider and model. Switching provider or model within a +Session does not lose conversational history — Smriti stores Turns independently of +any provider session. + +### Checkpoint + +A Checkpoint is a structured, immutable state snapshot. It contains: + +| Field | Description | +|---|---| +| **Title** | 3–5 word label for the state | +| **Objective** | The goal being worked toward at this point | +| **Summary** | Narrative of what was figured out | +| **Decisions** | Explicit choices made — only what was stated, not inferred | +| **Tasks** | Concrete action items identified | +| **Open Questions** | Unresolved questions at this point | +| **Entities** | Key concepts, tools, systems, or proper nouns | + +Checkpoints are created manually at meaningful points — before switching models, +before stepping away, when a significant decision is reached. The **Draft with AI** +feature uses a background intelligence model to extract a draft from the current +session's active Turns; the user reviews and saves. + +A Checkpoint is never created automatically. This is intentional: automatic +checkpointing produces noise, not signal. + +### Turn + +A Turn is a single unit in the event stream: one user message or one assistant reply, +with its provider, model, and sequence number recorded. Turns are append-only and +never edited. + +Turns are the raw material. Checkpoints are the distillate. + +--- + +## The Context Modes + +| Mode | What the model sees | +|---|---| +| **FRESH** | No Checkpoint context. Blank slate. Only the current Turn. | +| **HEAD** | The latest Checkpoint in the attached Space, plus recent Turns in this Session. | +| **MOUNTED** | A specific Checkpoint, plus only Turns created after mounting. Nothing else. | +| **FORKED** | The fork-source Checkpoint as base context, plus Turns created in this fork session. | + +The MOUNTED mode is the key differentiator. + +When you mount a specific Checkpoint, Smriti records the sequence number of the last +Turn at the moment of mounting. All subsequent requests pass only Turns with sequence +numbers above that boundary. Turns from other sessions, from other providers, or from +work that happened after that Checkpoint was created — none of it enters the context. +The isolation is enforced at the data layer, not by prompt instruction. + +--- + +## How It Works + +### Starting a session + +Open the workspace. A new Session is created automatically. By default it has no +Space attached (FRESH mode). + +### Attaching a Space + +Click the Space button in the thread header. Select an existing Space or create one. +Choose a memory scope: + +- **Latest checkpoint only** — the most recent Checkpoint provides base context +- **Latest 3 checkpoints** — the three most recent Checkpoints provide layered context + +### Creating a Checkpoint + +When you reach a meaningful point — a decision, a model switch, the end of a work +block — click the Checkpoint button in the thread header. + +The Checkpoint form opens with empty fields. Click **Draft with AI** to have the +background intelligence model extract a draft from the current session's active Turns. +The draft reflects only the active context mode (FRESH, HEAD, or MOUNTED). Review, +edit, and save. + +### Mounting a Checkpoint + +Open the Checkpoint history panel by clicking the context badge in the thread header. +Find the Checkpoint you want to return to. Click **Mount**. + +The header updates to show `MOUNTED · `. Any Turn you send from this point is +resolved against that Checkpoint's state only. + +To return to HEAD mode, click **Unmount** in the history panel. + +### Forking a Session + +Open the Branch Tree for a Space. Click **Fork** on any Checkpoint. Give the branch +a name. A new Session is created, seeded from that Checkpoint's state, with a clean +Turn history. + +The fork session starts in FORKED mode. Its reasoning develops independently from the +main branch. Both branches remain live in the same Space. + +### Comparing Checkpoints + +In the Branch Tree, select any two Checkpoints across any branches and click +**Compare**. The diff view shows decisions, tasks, and summaries side by side with +per-branch and shared items clearly marked. + +### Switching providers + +Change the provider and model in the session toolbar. The Session continues +uninterrupted. Smriti passes the same Checkpoint context and Turn history to the new +provider. No re-explanation required. + +--- + +## Current Limitations + +- Single-user; no authentication +- No merging of divergent Checkpoint lines +- No streaming responses — each Turn is a synchronous request/response cycle +- Transcript ingestion via paste (V1 API) is a legacy feature; not the primary workflow +- No mobile UI +- No MCP or browser extension integrations + +--- + +## Setup + +### Prerequisites + +- Python 3.11+ +- Node 18+ +- PostgreSQL 14+ +- At least one LLM provider API key (OpenAI, Anthropic, or OpenRouter), or use Mock + Mode to run without keys + +### Quick start + +```bash +git clone https://github.com/your-org/smriti +cd smriti + +# Copy environment template +cp .env.example .env + +# Install all dependencies and run database migrations +make setup + +# Start backend (terminal 1) +make dev + +# Start frontend (terminal 2) +make dev-frontend +``` + +Frontend: `http://localhost:5173` — Backend API: `http://localhost:8000` + +### Docker + +```bash +make up # Start all services (postgres + backend + frontend) +make logs # Follow logs +make down # Stop all services +``` + +--- + +## Provider Configuration + +Smriti uses two independent provider slots: + +**Chat provider** — the model you converse with. Selected per-session in the UI +toolbar. Supports: OpenAI, Anthropic, OpenRouter. + +**Background intelligence** — a separate model used for Draft with AI and session +auto-titling. Configured server-side. Not visible in the chat UI. + +### YAML configuration (recommended) + +Copy `backend/config/providers.example.yaml` to `backend/config/providers.yaml` and +fill in your keys: + +```yaml +providers: + openai: + api_key: "sk-..." + default_model: "gpt-4o" + + anthropic: + api_key: "sk-ant-..." + default_model: "claude-sonnet-4-6" + + openrouter: + api_key: "sk-or-..." + base_url: "https://openrouter.ai/api/v1" + +chat: + default_provider: "openrouter" + +background_intelligence: + provider: "openai" + model: "gpt-4o-mini" +``` + +### Environment variables + +```bash +OPENAI_API_KEY=sk-... +ANTHROPIC_API_KEY=sk-ant-... +OPENROUTER_API_KEY=sk-or-... +SMRITI_DEFAULT_PROVIDER=openrouter +DATABASE_URL=postgresql://smriti:smriti@localhost:5432/smriti +``` + +Environment variables take precedence over YAML values. API keys are never returned +by any API endpoint. + +### Mock Mode + +The UI includes a **Mock Mode** toggle (compose bar) that uses a deterministic mock +adapter — no API calls, scripted responses. Useful for testing checkpoint and session +mechanics without a live provider key. + +--- + +## Try the Demo + +The `demos/branching-reasoning-demo/` directory contains a complete, repeatable demo +scenario demonstrating Smriti's branching-reasoning workflow. It includes: + +- Step-by-step runbook +- Exact messages to paste for each turn +- Expected diff output +- Presenter talk track + +See [demos/branching-reasoning-demo/README.md](demos/branching-reasoning-demo/README.md) +for the full walkthrough. + +--- + +## Technical Stack + +| Layer | Technology | +|---|---| +| Backend | FastAPI, SQLAlchemy, Alembic, Python 3.11+ | +| Database | PostgreSQL | +| Frontend | React 18, TypeScript, Vite, Tailwind CSS | +| Provider adapters | OpenAI SDK, Anthropic SDK, OpenRouter (OpenAI-compatible) | + +The backend API is versioned by product generation. V4 handles chat sessions and +message sending. V5 handles checkpoint and lineage operations. V1 and V2 are legacy +endpoints retained for compatibility but not part of the current primary workflow. + +--- + +## Roadmap + +- Streaming responses +- Multi-user Spaces with authentication +- Provider expansion (Gemini, local models via Ollama) +- Source Turn range recorded on Checkpoints (which conversation produced this snapshot) +- MCP integrations +- Checkpoint merging + +--- + +## Further Reading + +- [ARCHITECTURE.md](ARCHITECTURE.md) — system model, checkpoint isolation mechanism, + provider abstraction, API versioning +- [docs/API.md](docs/API.md) — endpoint reference for V4 (chat) and V5 (checkpoint + and lineage) +- [DECISIONS.md](DECISIONS.md) — key architectural and product decisions +- [CONTRIBUTING.md](CONTRIBUTING.md) — how to set up the dev environment and contribute diff --git a/REPO_STRUCTURE.md b/REPO_STRUCTURE.md new file mode 100644 index 0000000..045e981 --- /dev/null +++ b/REPO_STRUCTURE.md @@ -0,0 +1,97 @@ +# Smriti — Repository Structure + +``` +smriti/ +│ +├── README.md Product overview, core concepts, setup +├── ARCHITECTURE.md System model, isolation mechanism, API versioning +├── DECISIONS.md Key architectural and product decisions +├── CONTRIBUTING.md Development setup and contribution guide +├── REPO_STRUCTURE.md This file +│ +├── Makefile Dev, test, build, and migration targets +├── docker-compose.yml Postgres + backend + frontend services +├── .env.example Environment variable template +│ +├── backend/ +│ ├── app/ +│ │ ├── main.py FastAPI app factory, CORS, router registration +│ │ ├── config.py Pydantic settings (DATABASE_URL, DEBUG) +│ │ ├── config_loader.py Provider config loader (providers.yaml + env vars) +│ │ ├── schemas/ +│ │ │ └── __init__.py Pydantic request/response schemas +│ │ ├── db/ +│ │ │ ├── database.py SQLAlchemy engine and session factory +│ │ │ └── models.py ORM models: RepoModel, CommitModel, ChatSession, +│ │ │ TurnEvent +│ │ ├── domain/ +│ │ │ └── enums.py SessionStatus, TargetTool, etc. +│ │ ├── api/ +│ │ │ └── routes/ +│ │ │ ├── chat.py V4: sessions, send_message, commit, head, +│ │ │ │ provider status +│ │ │ ├── checkpoint.py V5: draft_checkpoint +│ │ │ ├── lineage.py V5: fork, branch tree, checkpoint compare, +│ │ │ │ reachable checkpoints +│ │ │ ├── repos.py V2: Space CRUD, Checkpoint CRUD +│ │ │ └── sessions.py V1: transcript ingestion (legacy) +│ │ └── providers/ +│ │ ├── registry.py Provider lookup and adapter instantiation +│ │ ├── openai_adapter.py +│ │ ├── anthropic_adapter.py +│ │ └── mock_adapter.py Deterministic mock for testing +│ ├── config/ +│ │ ├── providers.example.yaml Template — copy to providers.yaml +│ │ └── providers.yaml Your keys (gitignored, not committed) +│ ├── alembic/ Database migrations +│ ├── tests/ Backend tests +│ └── pyproject.toml Python dependencies +│ +├── frontend/ +│ ├── src/ +│ │ ├── pages/ +│ │ │ ├── ChatWorkspacePage.tsx Primary UI: chat, sidebar, checkpoint +│ │ │ │ modal, history panel, context indicators +│ │ │ └── LineagePage.tsx Branch tree, checkpoint compare +│ │ ├── api/ +│ │ │ └── client.ts API client functions (V4, V5, V2) +│ │ ├── types/ +│ │ │ └── index.ts TypeScript type definitions +│ │ └── main.tsx App entry point, router +│ └── package.json +│ +├── docs/ +│ └── API.md V4 and V5 endpoint reference +│ +└── demos/ + └── branching-reasoning-demo/ Complete demo scenario with runbook, + script, and expected outcomes +``` + +--- + +## API version map + +| Prefix | Module | Status | Notes | +|---|---|---|---| +| `/api/v1` | `sessions.py` | Legacy | Transcript paste ingestion | +| `/api/v2` | `repos.py` | Partially current | Space and Checkpoint CRUD used by UI; agent-push workflow is legacy | +| `/api/v4` | `chat.py` | Current | Primary chat and session API | +| `/api/v5` | `checkpoint.py`, `lineage.py` | Current | Checkpoint draft, fork, lineage | + +--- + +## Make targets + +``` +make setup Install deps + run migrations (first-time setup) +make dev Run backend dev server (port 8000) +make dev-frontend Run frontend dev server (port 5173) +make up Start all services via Docker Compose +make down Stop all services +make test Run all backend tests +make lint Lint backend code (ruff) +make format Format backend code (ruff) +make migrate Run pending Alembic migrations +make migration Create a new migration (usage: make migration msg="...") +``` diff --git a/SECURITY.md b/SECURITY.md new file mode 100644 index 0000000..a76564e --- /dev/null +++ b/SECURITY.md @@ -0,0 +1,43 @@ +# Security + +## Reporting a vulnerability + +Please do not report security vulnerabilities through public GitHub issues. + +Email the maintainers directly (see repository contact info) or open a private +security advisory via GitHub's **Security → Advisories** tab. + +Include: +- Description of the issue +- Steps to reproduce +- Affected version (git hash or tag) +- Potential impact + +We will acknowledge reports within 72 hours and aim to resolve confirmed +vulnerabilities within 14 days. + +--- + +## Known security boundaries + +Smriti is a single-user tool with no authentication layer. It is designed for +local or private-network deployment. + +**Do not expose the backend port (8000) to the public internet.** The API has no +authentication; anyone who can reach it can read and write all data. + +If you deploy Smriti on a server: + +- Put the backend behind a reverse proxy (nginx, Caddy, etc.) +- Restrict access with network-level controls or auth middleware +- Change the default database credentials in `docker-compose.yml` and `.env` +- Do not commit `backend/config/providers.yaml` — it is gitignored for this reason + +--- + +## API keys + +API keys are configured via environment variables or `backend/config/providers.yaml` +(gitignored). They are never returned by any API endpoint. The backend logging +middleware applies a `SecretGuardFilter` that redacts `sk-*` patterns from all log +output. diff --git a/backend/Dockerfile b/backend/Dockerfile new file mode 100644 index 0000000..fca14da --- /dev/null +++ b/backend/Dockerfile @@ -0,0 +1,22 @@ +FROM python:3.11-slim-bookworm + +WORKDIR /app + +# Install system dependencies +RUN apt-get update && apt-get install -y --no-install-recommends \ + gcc libpq-dev && \ + rm -rf /var/lib/apt/lists/* + +# Copy all source first (needed for pip install) +COPY pyproject.toml . +COPY app/ app/ + +# Install Python dependencies (non-editable) +RUN pip install --no-cache-dir . + +# Copy remaining files (alembic, tests, etc.) +COPY . . + +EXPOSE 8000 + +CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000"] diff --git a/backend/alembic.ini b/backend/alembic.ini new file mode 100644 index 0000000..cb24ccc --- /dev/null +++ b/backend/alembic.ini @@ -0,0 +1,77 @@ +# A generic, async-compatible Alembic configuration file. +[alembic] +# path to migration scripts +script_location = alembic + +# template used to generate migration file names +file_template = %%(year)d_%%(month).2d_%%(day).2d_%%(hour).2d%%(minute).2d_%%(rev)s_%%(slug)s + +# sys.path, prepended +prepend_sys_path = . + +# timezone +# timezone = + +# max length of characters to apply to the "slug" field +# truncate_slug_length = 40 + +# set to 'true' to run the environment during +# the 'revision' command, regardless of autogenerate +# revision_environment = false + +# set to 'true' to allow .pyc and .pyo files without +# having a matching .py file be detected as revisions +# sourceless = false + +# version location specification; +# version_path_separator = os +# version_locations = %(here)s/bar:%(here)s/bat:alembic/versions + +# set to 'true' to search source files in +# the version_locations, recursively +# recursive_version_locations = false + +# the output encoding used by revision files +# output_encoding = utf-8 + +sqlalchemy.url = postgresql://smriti:smriti@localhost:5432/smriti + + +[post_write_hooks] +# post_write_hooks defines scripts or Python functions that are run +# on newly generated revision scripts. + + +[loggers] +keys = root,sqlalchemy,alembic + +[handlers] +keys = console + +[formatters] +keys = generic + +[logger_root] +level = WARN +handlers = console +qualname = + +[logger_sqlalchemy] +level = WARN +handlers = +qualname = sqlalchemy.engine + +[logger_alembic] +level = INFO +handlers = +qualname = alembic + +[handler_console] +class = StreamHandler +args = (sys.stderr,) +level = NOTSET +formatter = generic + +[formatter_generic] +format = %(levelname)-5.5s [%(name)s] %(message)s +datefmt = %H:%M:%S diff --git a/backend/alembic/env.py b/backend/alembic/env.py new file mode 100644 index 0000000..110f7e6 --- /dev/null +++ b/backend/alembic/env.py @@ -0,0 +1,58 @@ +from logging.config import fileConfig + +from alembic import context +from sqlalchemy import engine_from_config, pool + +from app.config import settings +from app.db.database import Base +from app.db.models import ( # noqa: F401 — import to register models + ContextPackModel, + ExtractionResultModel, + MessageModel, + SessionModel, +) + +# this is the Alembic Config object +config = context.config + +# Override sqlalchemy.url from settings +config.set_main_option("sqlalchemy.url", settings.database_url) + +# Interpret the config file for Python logging +if config.config_file_name is not None: + fileConfig(config.config_file_name) + +# target metadata for autogenerate +target_metadata = Base.metadata + + +def run_migrations_offline() -> None: + """Run migrations in 'offline' mode.""" + url = config.get_main_option("sqlalchemy.url") + context.configure( + url=url, + target_metadata=target_metadata, + literal_binds=True, + dialect_opts={"paramstyle": "named"}, + ) + with context.begin_transaction(): + context.run_migrations() + + +def run_migrations_online() -> None: + """Run migrations in 'online' mode.""" + connectable = engine_from_config( + config.get_section(config.config_ini_section, {}), + prefix="sqlalchemy.", + poolclass=pool.NullPool, + ) + with connectable.connect() as connection: + context.configure(connection=connection, target_metadata=target_metadata) + with context.begin_transaction(): + context.run_migrations() + + +if context.is_offline_mode(): + run_migrations_offline() +else: + run_migrations_online() diff --git a/backend/alembic/script.py.mako b/backend/alembic/script.py.mako new file mode 100644 index 0000000..fbc4b07 --- /dev/null +++ b/backend/alembic/script.py.mako @@ -0,0 +1,26 @@ +"""${message} + +Revision ID: ${up_revision} +Revises: ${down_revision | comma,n} +Create Date: ${create_date} + +""" +from typing import Sequence, Union + +from alembic import op +import sqlalchemy as sa +${imports if imports else ""} + +# revision identifiers, used by Alembic. +revision: str = ${repr(up_revision)} +down_revision: Union[str, None] = ${repr(down_revision)} +branch_labels: Union[str, Sequence[str], None] = ${repr(branch_labels)} +depends_on: Union[str, Sequence[str], None] = ${repr(depends_on)} + + +def upgrade() -> None: + ${upgrades if upgrades else "pass"} + + +def downgrade() -> None: + ${downgrades if downgrades else "pass"} diff --git a/backend/alembic/versions/.gitkeep b/backend/alembic/versions/.gitkeep new file mode 100644 index 0000000..e69de29 diff --git a/backend/alembic/versions/2026_03_19_2114_393755616947_initial_tables.py b/backend/alembic/versions/2026_03_19_2114_393755616947_initial_tables.py new file mode 100644 index 0000000..77ebf9f --- /dev/null +++ b/backend/alembic/versions/2026_03_19_2114_393755616947_initial_tables.py @@ -0,0 +1,76 @@ +"""initial_tables + +Revision ID: 393755616947 +Revises: +Create Date: 2026-03-19 21:14:02.416570 + +""" +from typing import Sequence, Union + +from alembic import op +import sqlalchemy as sa +from sqlalchemy.dialects import postgresql + +# revision identifiers, used by Alembic. +revision: str = '393755616947' +down_revision: Union[str, None] = None +branch_labels: Union[str, Sequence[str], None] = None +depends_on: Union[str, Sequence[str], None] = None + + +def upgrade() -> None: + # ### commands auto generated by Alembic - please adjust! ### + op.create_table('sessions', + sa.Column('id', sa.UUID(), nullable=False), + sa.Column('title', sa.String(length=255), nullable=True), + sa.Column('source_tool', sa.String(length=50), nullable=True), + sa.Column('raw_transcript', sa.Text(), nullable=False), + sa.Column('status', sa.String(length=20), nullable=False), + sa.Column('created_at', sa.DateTime(timezone=True), nullable=False), + sa.Column('updated_at', sa.DateTime(timezone=True), nullable=False), + sa.PrimaryKeyConstraint('id') + ) + op.create_table('context_packs', + sa.Column('id', sa.UUID(), nullable=False), + sa.Column('session_id', sa.UUID(), nullable=False), + sa.Column('target_tool', sa.String(length=20), nullable=False), + sa.Column('content', sa.Text(), nullable=False), + sa.Column('format', sa.String(length=20), nullable=False), + sa.Column('created_at', sa.DateTime(timezone=True), nullable=False), + sa.ForeignKeyConstraint(['session_id'], ['sessions.id'], ondelete='CASCADE'), + sa.PrimaryKeyConstraint('id') + ) + op.create_table('extraction_results', + sa.Column('id', sa.UUID(), nullable=False), + sa.Column('session_id', sa.UUID(), nullable=False), + sa.Column('summary', sa.Text(), nullable=False), + sa.Column('decisions', postgresql.JSONB(astext_type=sa.Text()), nullable=False), + sa.Column('tasks', postgresql.JSONB(astext_type=sa.Text()), nullable=False), + sa.Column('open_questions', postgresql.JSONB(astext_type=sa.Text()), nullable=False), + sa.Column('entities', postgresql.JSONB(astext_type=sa.Text()), nullable=False), + sa.Column('code_snippets', postgresql.JSONB(astext_type=sa.Text()), nullable=False), + sa.Column('created_at', sa.DateTime(timezone=True), nullable=False), + sa.ForeignKeyConstraint(['session_id'], ['sessions.id'], ondelete='CASCADE'), + sa.PrimaryKeyConstraint('id'), + sa.UniqueConstraint('session_id') + ) + op.create_table('messages', + sa.Column('id', sa.UUID(), nullable=False), + sa.Column('session_id', sa.UUID(), nullable=False), + sa.Column('role', sa.String(length=20), nullable=False), + sa.Column('content', sa.Text(), nullable=False), + sa.Column('position', sa.Integer(), nullable=False), + sa.Column('created_at', sa.DateTime(timezone=True), nullable=False), + sa.ForeignKeyConstraint(['session_id'], ['sessions.id'], ondelete='CASCADE'), + sa.PrimaryKeyConstraint('id') + ) + # ### end Alembic commands ### + + +def downgrade() -> None: + # ### commands auto generated by Alembic - please adjust! ### + op.drop_table('messages') + op.drop_table('extraction_results') + op.drop_table('context_packs') + op.drop_table('sessions') + # ### end Alembic commands ### diff --git a/backend/alembic/versions/2026_03_19_2258_fab0a53299d6_add_memories_table_and_pgvector.py b/backend/alembic/versions/2026_03_19_2258_fab0a53299d6_add_memories_table_and_pgvector.py new file mode 100644 index 0000000..b26323b --- /dev/null +++ b/backend/alembic/versions/2026_03_19_2258_fab0a53299d6_add_memories_table_and_pgvector.py @@ -0,0 +1,48 @@ +"""add memories table and pgvector + +Revision ID: fab0a53299d6 +Revises: 393755616947 +Create Date: 2026-03-19 22:58:53.187731 + +""" +from typing import Sequence, Union + +from alembic import op +import sqlalchemy as sa +from sqlalchemy.dialects import postgresql +import pgvector.sqlalchemy + +# revision identifiers, used by Alembic. +revision: str = 'fab0a53299d6' +down_revision: Union[str, None] = '393755616947' +branch_labels: Union[str, Sequence[str], None] = None +depends_on: Union[str, Sequence[str], None] = None + + +def upgrade() -> None: + # ### commands auto generated by Alembic - please adjust! ### + op.execute('CREATE EXTENSION IF NOT EXISTS vector') + op.create_table('memories', + sa.Column('id', sa.UUID(), nullable=False), + sa.Column('user_id', sa.UUID(), nullable=False), + sa.Column('type', sa.String(length=50), nullable=False), + sa.Column('content', sa.Text(), nullable=False), + sa.Column('embedding', pgvector.sqlalchemy.vector.VECTOR(dim=1536), nullable=True), + sa.Column('source', sa.String(length=255), nullable=True), + sa.Column('confidence', sa.Float(), nullable=False), + sa.Column('importance', sa.Float(), nullable=False), + sa.Column('status', sa.String(length=50), nullable=False), + sa.Column('metadata', postgresql.JSONB(astext_type=sa.Text()), nullable=False), + sa.Column('created_at', sa.DateTime(timezone=True), nullable=False), + sa.Column('updated_at', sa.DateTime(timezone=True), nullable=False), + sa.PrimaryKeyConstraint('id') + ) + op.create_index(op.f('ix_memories_user_id'), 'memories', ['user_id'], unique=False) + # ### end Alembic commands ### + + +def downgrade() -> None: + # ### commands auto generated by Alembic - please adjust! ### + op.drop_index(op.f('ix_memories_user_id'), table_name='memories') + op.drop_table('memories') + # ### end Alembic commands ### diff --git a/backend/alembic/versions/2026_03_20_0014_c80fc0e31e4a_add_repo_commit_models.py b/backend/alembic/versions/2026_03_20_0014_c80fc0e31e4a_add_repo_commit_models.py new file mode 100644 index 0000000..22049db --- /dev/null +++ b/backend/alembic/versions/2026_03_20_0014_c80fc0e31e4a_add_repo_commit_models.py @@ -0,0 +1,68 @@ +"""add_repo_commit_models + +Revision ID: c80fc0e31e4a +Revises: fab0a53299d6 +Create Date: 2026-03-20 00:14:47.622498 + +""" +from typing import Sequence, Union + +from alembic import op +import sqlalchemy as sa +from sqlalchemy.dialects import postgresql + +# revision identifiers, used by Alembic. +revision: str = 'c80fc0e31e4a' +down_revision: Union[str, None] = 'fab0a53299d6' +branch_labels: Union[str, Sequence[str], None] = None +depends_on: Union[str, Sequence[str], None] = None + + +def upgrade() -> None: + # ### commands auto generated by Alembic - please adjust! ### + op.create_table('repos', + sa.Column('id', sa.UUID(), nullable=False), + sa.Column('repo_slug', sa.String(length=255), nullable=True), + sa.Column('name', sa.String(length=255), nullable=False), + sa.Column('description', sa.Text(), nullable=False), + sa.Column('user_id', sa.UUID(), nullable=False), + sa.Column('metadata', postgresql.JSONB(astext_type=sa.Text()), nullable=False), + sa.Column('created_at', sa.DateTime(timezone=True), nullable=False), + sa.Column('updated_at', sa.DateTime(timezone=True), nullable=False), + sa.PrimaryKeyConstraint('id'), + sa.UniqueConstraint('repo_slug') + ) + op.create_index(op.f('ix_repos_user_id'), 'repos', ['user_id'], unique=False) + op.create_table('commits', + sa.Column('id', sa.UUID(), nullable=False), + sa.Column('repo_id', sa.UUID(), nullable=False), + sa.Column('commit_hash', sa.String(length=64), nullable=False), + sa.Column('parent_commit_id', sa.UUID(), nullable=True), + sa.Column('branch_name', sa.String(length=255), nullable=False), + sa.Column('author_agent', sa.String(length=255), nullable=True), + sa.Column('author_type', sa.String(length=50), nullable=False), + sa.Column('message', sa.String(length=255), nullable=False), + sa.Column('summary', sa.Text(), nullable=False), + sa.Column('objective', sa.Text(), nullable=False), + sa.Column('decisions', postgresql.JSONB(astext_type=sa.Text()), nullable=False), + sa.Column('tasks', postgresql.JSONB(astext_type=sa.Text()), nullable=False), + sa.Column('open_questions', postgresql.JSONB(astext_type=sa.Text()), nullable=False), + sa.Column('entities', postgresql.JSONB(astext_type=sa.Text()), nullable=False), + sa.Column('context_blob', postgresql.JSONB(astext_type=sa.Text()), nullable=False), + sa.Column('raw_source_text', sa.Text(), nullable=True), + sa.Column('metadata', postgresql.JSONB(astext_type=sa.Text()), nullable=False), + sa.Column('created_at', sa.DateTime(timezone=True), nullable=False), + sa.ForeignKeyConstraint(['parent_commit_id'], ['commits.id'], ondelete='SET NULL'), + sa.ForeignKeyConstraint(['repo_id'], ['repos.id'], ondelete='CASCADE'), + sa.PrimaryKeyConstraint('id'), + sa.UniqueConstraint('commit_hash') + ) + # ### end Alembic commands ### + + +def downgrade() -> None: + # ### commands auto generated by Alembic - please adjust! ### + op.drop_table('commits') + op.drop_index(op.f('ix_repos_user_id'), table_name='repos') + op.drop_table('repos') + # ### end Alembic commands ### diff --git a/backend/alembic/versions/2026_03_20_1122_65900af1312b_add_chat_session_turn_event.py b/backend/alembic/versions/2026_03_20_1122_65900af1312b_add_chat_session_turn_event.py new file mode 100644 index 0000000..405e337 --- /dev/null +++ b/backend/alembic/versions/2026_03_20_1122_65900af1312b_add_chat_session_turn_event.py @@ -0,0 +1,67 @@ +"""add_chat_session_turn_event + +Revision ID: 65900af1312b +Revises: c80fc0e31e4a +Create Date: 2026-03-20 11:22:37.745155 + +""" +from typing import Sequence, Union + +from alembic import op +import sqlalchemy as sa +from sqlalchemy.dialects import postgresql + +# revision identifiers, used by Alembic. +revision: str = '65900af1312b' +down_revision: Union[str, None] = 'c80fc0e31e4a' +branch_labels: Union[str, Sequence[str], None] = None +depends_on: Union[str, Sequence[str], None] = None + + +def upgrade() -> None: + # ### commands auto generated by Alembic - please adjust! ### + op.create_table('chat_sessions', + sa.Column('id', sa.UUID(), nullable=False), + sa.Column('repo_id', sa.UUID(), nullable=False), + sa.Column('title', sa.String(length=255), nullable=False), + sa.Column('active_provider', sa.String(length=50), nullable=False), + sa.Column('active_model', sa.String(length=255), nullable=False), + sa.Column('seeded_commit_id', sa.UUID(), nullable=True), + sa.Column('metadata', postgresql.JSONB(astext_type=sa.Text()), nullable=False), + sa.Column('created_at', sa.DateTime(timezone=True), nullable=False), + sa.Column('updated_at', sa.DateTime(timezone=True), nullable=False), + sa.ForeignKeyConstraint(['repo_id'], ['repos.id'], ondelete='CASCADE'), + sa.ForeignKeyConstraint(['seeded_commit_id'], ['commits.id'], ondelete='SET NULL'), + sa.PrimaryKeyConstraint('id') + ) + op.create_index(op.f('ix_chat_sessions_repo_id'), 'chat_sessions', ['repo_id'], unique=False) + op.create_table('turn_events', + sa.Column('id', sa.UUID(), nullable=False), + sa.Column('session_id', sa.UUID(), nullable=False), + sa.Column('repo_id', sa.UUID(), nullable=False), + sa.Column('role', sa.String(length=20), nullable=False), + sa.Column('content', sa.Text(), nullable=False), + sa.Column('provider', sa.String(length=50), nullable=False), + sa.Column('model', sa.String(length=255), nullable=False), + sa.Column('sequence_number', sa.Integer(), nullable=False), + sa.Column('commit_id', sa.UUID(), nullable=True), + sa.Column('metadata', postgresql.JSONB(astext_type=sa.Text()), nullable=False), + sa.Column('created_at', sa.DateTime(timezone=True), nullable=False), + sa.ForeignKeyConstraint(['commit_id'], ['commits.id'], ondelete='SET NULL'), + sa.ForeignKeyConstraint(['repo_id'], ['repos.id'], ondelete='CASCADE'), + sa.ForeignKeyConstraint(['session_id'], ['chat_sessions.id'], ondelete='CASCADE'), + sa.PrimaryKeyConstraint('id') + ) + op.create_index(op.f('ix_turn_events_repo_id'), 'turn_events', ['repo_id'], unique=False) + op.create_index(op.f('ix_turn_events_session_id'), 'turn_events', ['session_id'], unique=False) + # ### end Alembic commands ### + + +def downgrade() -> None: + # ### commands auto generated by Alembic - please adjust! ### + op.drop_index(op.f('ix_turn_events_session_id'), table_name='turn_events') + op.drop_index(op.f('ix_turn_events_repo_id'), table_name='turn_events') + op.drop_table('turn_events') + op.drop_index(op.f('ix_chat_sessions_repo_id'), table_name='chat_sessions') + op.drop_table('chat_sessions') + # ### end Alembic commands ### diff --git a/backend/alembic/versions/2026_03_21_0046_8473b83ba7a7_make_repo_id_nullable.py b/backend/alembic/versions/2026_03_21_0046_8473b83ba7a7_make_repo_id_nullable.py new file mode 100644 index 0000000..30f681c --- /dev/null +++ b/backend/alembic/versions/2026_03_21_0046_8473b83ba7a7_make_repo_id_nullable.py @@ -0,0 +1,40 @@ +"""make_repo_id_nullable + +Revision ID: 8473b83ba7a7 +Revises: 65900af1312b +Create Date: 2026-03-21 00:46:04.456225 + +""" +from typing import Sequence, Union + +from alembic import op +import sqlalchemy as sa + + +# revision identifiers, used by Alembic. +revision: str = '8473b83ba7a7' +down_revision: Union[str, None] = '65900af1312b' +branch_labels: Union[str, Sequence[str], None] = None +depends_on: Union[str, Sequence[str], None] = None + + +def upgrade() -> None: + # ### commands auto generated by Alembic - please adjust! ### + op.alter_column('chat_sessions', 'repo_id', + existing_type=sa.UUID(), + nullable=True) + op.alter_column('turn_events', 'repo_id', + existing_type=sa.UUID(), + nullable=True) + # ### end Alembic commands ### + + +def downgrade() -> None: + # ### commands auto generated by Alembic - please adjust! ### + op.alter_column('turn_events', 'repo_id', + existing_type=sa.UUID(), + nullable=False) + op.alter_column('chat_sessions', 'repo_id', + existing_type=sa.UUID(), + nullable=False) + # ### end Alembic commands ### diff --git a/backend/alembic/versions/2026_03_21_1200_a1b2c3d4e5f6_add_fork_fields_to_chat_sessions.py b/backend/alembic/versions/2026_03_21_1200_a1b2c3d4e5f6_add_fork_fields_to_chat_sessions.py new file mode 100644 index 0000000..cda5d80 --- /dev/null +++ b/backend/alembic/versions/2026_03_21_1200_a1b2c3d4e5f6_add_fork_fields_to_chat_sessions.py @@ -0,0 +1,41 @@ +"""add_fork_fields_to_chat_sessions + +Revision ID: a1b2c3d4e5f6 +Revises: 8473b83ba7a7 +Create Date: 2026-03-21 12:00:00.000000 + +""" +from typing import Sequence, Union + +from alembic import op +import sqlalchemy as sa + + +# revision identifiers, used by Alembic. +revision: str = 'a1b2c3d4e5f6' +down_revision: Union[str, None] = '8473b83ba7a7' +branch_labels: Union[str, Sequence[str], None] = None +depends_on: Union[str, Sequence[str], None] = None + + +def upgrade() -> None: + op.add_column( + 'chat_sessions', + sa.Column('forked_from_checkpoint_id', sa.UUID(), nullable=True), + ) + op.add_column( + 'chat_sessions', + sa.Column('branch_name', sa.String(length=255), nullable=False, server_default='main'), + ) + op.create_foreign_key( + 'fk_chat_sessions_forked_from_checkpoint_id', + 'chat_sessions', 'commits', + ['forked_from_checkpoint_id'], ['id'], + ondelete='SET NULL', + ) + + +def downgrade() -> None: + op.drop_constraint('fk_chat_sessions_forked_from_checkpoint_id', 'chat_sessions', type_='foreignkey') + op.drop_column('chat_sessions', 'branch_name') + op.drop_column('chat_sessions', 'forked_from_checkpoint_id') diff --git a/backend/app/__init__.py b/backend/app/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/backend/app/api/__init__.py b/backend/app/api/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/backend/app/api/deps.py b/backend/app/api/deps.py new file mode 100644 index 0000000..6be51e0 --- /dev/null +++ b/backend/app/api/deps.py @@ -0,0 +1,31 @@ +"""API dependency injection.""" + +from app.config import settings +from app.services.extractor import ExtractorService +from app.services.llm.mock_provider import MockProvider +from app.services.llm.openai_provider import OpenAIProvider +from app.services.embedding import EmbeddingService, MockEmbeddingService, OpenAIEmbeddingService + + +def get_extractor_service() -> ExtractorService: + """Get the extraction service with the appropriate LLM provider. + + Uses MockProvider if no OpenAI API key is configured, + otherwise uses OpenAIProvider. + """ + if settings.openai_api_key: + provider = OpenAIProvider() + else: + provider = MockProvider() + return ExtractorService(llm_provider=provider) + + +def get_embedding_service() -> EmbeddingService: + """Get the embedding service. + + Uses MockEmbeddingService if no OpenAI API key is configured, + otherwise uses OpenAIEmbeddingService. + """ + if settings.openai_api_key: + return OpenAIEmbeddingService(api_key=settings.openai_api_key) + return MockEmbeddingService() diff --git a/backend/app/api/routes/__init__.py b/backend/app/api/routes/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/backend/app/api/routes/chat.py b/backend/app/api/routes/chat.py new file mode 100644 index 0000000..a0b0fb3 --- /dev/null +++ b/backend/app/api/routes/chat.py @@ -0,0 +1,671 @@ +""" +V4 Chat API routes. + +Session lifecycle: + POST /api/v4/chat/spaces/{repo_id}/sessions – create session (seeds context from head commit) + GET /api/v4/chat/spaces/{repo_id}/sessions/{sid} – get session + GET /api/v4/chat/spaces/{repo_id}/sessions/{sid}/turns – list turns + +Conversation: + POST /api/v4/chat/send – send a user message, get assistant reply + POST /api/v4/chat/commit – manually commit session state as a Commit + +Space head: + GET /api/v4/chat/spaces/{repo_id}/head – latest commit + latest session + +Provider status: + GET /api/v4/chat/providers – list provider config status +""" + +from __future__ import annotations + +import hashlib +import json +import uuid +from datetime import datetime, timezone +from typing import Optional + +from fastapi import APIRouter, Depends, HTTPException +from pydantic import BaseModel, Field +from sqlalchemy import select +from sqlalchemy.orm import Session + +from app.db.database import get_db +from app.db.models import ChatSession, CommitModel, RepoModel, TurnEvent +from app.config_loader import get_config, providers_status, ProviderNotConfiguredError +from app.providers.registry import get_adapter, get_mock_adapter + +router = APIRouter(prefix="/chat", tags=["chat-v4"]) + +DEMO_USER_ID = uuid.UUID("00000000-0000-0000-0000-00000000000a") +MAX_CONTEXT_TURNS = 20 # how many recent turns to pass as conversation history + + +# ── Helpers ─────────────────────────────────────────────────────────────────── + +def _utcnow() -> datetime: + return datetime.now(timezone.utc) + + +def _get_repo(repo_id: uuid.UUID, db: Session) -> RepoModel: + repo = db.get(RepoModel, repo_id) if repo_id else None + if not repo or repo.user_id != DEMO_USER_ID: + raise HTTPException(status_code=404, detail="Space not found") + return repo + + +def _get_latest_commit(repo_id: uuid.UUID, db: Session) -> CommitModel | None: + stmt = ( + select(CommitModel) + .where(CommitModel.repo_id == repo_id, CommitModel.branch_name == "main") + .order_by(CommitModel.created_at.desc()) + .limit(1) + ) + return db.scalars(stmt).first() + + +def _get_latest_commit_on_branch(repo_id: uuid.UUID, branch: str, db: Session) -> CommitModel | None: + """Return the most recent commit on a specific branch within a repo.""" + stmt = ( + select(CommitModel) + .where(CommitModel.repo_id == repo_id, CommitModel.branch_name == branch) + .order_by(CommitModel.created_at.desc()) + .limit(1) + ) + return db.scalars(stmt).first() + + +def _get_checkpoints_for_scope(repo_id: uuid.UUID, db: Session, scope: str) -> list[CommitModel]: + """Return checkpoints based on memory scope (oldest-first).""" + n = 3 if scope == "latest_3" else 1 + stmt = ( + select(CommitModel) + .where(CommitModel.repo_id == repo_id, CommitModel.branch_name == "main") + .order_by(CommitModel.created_at.desc()) + .limit(n) + ) + return list(reversed(db.scalars(stmt).all())) # oldest first + + +def _walk_ancestors(commit: CommitModel, db: Session, n: int) -> list[CommitModel]: + """Walk parent chain and return up to n checkpoints, oldest-first.""" + chain = [commit] + current = commit + for _ in range(n - 1): + if not current.parent_commit_id: + break + parent = db.get(CommitModel, current.parent_commit_id) + if not parent: + break + chain.append(parent) + current = parent + return list(reversed(chain)) # oldest first + + +def _resolve_checkpoints( + repo_id: Optional[uuid.UUID], + mounted_checkpoint_id: Optional[str], + scope: str, + db: Session, +) -> list[CommitModel]: + """ + Resolve which checkpoints to mount for context building. + + Priority: + 1. If mounted_checkpoint_id is set, use that specific checkpoint as anchor. + - scope latest_1 → only that checkpoint + - scope latest_3 → that checkpoint + up to 2 ancestors + 2. Otherwise fall back to _get_checkpoints_for_scope (latest N from repo head). + """ + if mounted_checkpoint_id: + try: + commit = db.get(CommitModel, uuid.UUID(mounted_checkpoint_id)) + except (ValueError, Exception): + commit = None + if commit: + n = 3 if scope == "latest_3" else 1 + return _walk_ancestors(commit, db, n) + if repo_id: + return _get_checkpoints_for_scope(repo_id, db, scope) + return [] + + +def build_prompt_from_checkpoints(checkpoints: list[CommitModel], recent_messages: list[TurnEvent], user_input: str) -> str: + """Reconstructs the conversation context from Smriti memory (supports multiple checkpoints).""" + lines = ["You are continuing a conversation.\n"] + + for i, ckpt in enumerate(checkpoints): + label = f"Checkpoint {i + 1}" if len(checkpoints) > 1 else "Checkpoint" + if ckpt.summary: + lines.append(f"{label} Summary:") + lines.append(ckpt.summary + "\n") + if ckpt.decisions: + lines.append(f"{label} Key Decisions:") + for d in ckpt.decisions: + lines.append(f"- {d}") + lines.append("") + if ckpt.tasks: + lines.append(f"{label} Open Tasks:") + for t in ckpt.tasks: + lines.append(f"- {t}") + lines.append("") + + if recent_messages: + lines.append("Recent Conversation:") + for m in recent_messages: + role = "User" if m.role == "user" else "Assistant" + lines.append(f"{role}: {m.content}") + lines.append("") + + lines.append("User Query:") + lines.append(user_input) + + return "\n".join(lines) + + +# Keep single-checkpoint variant for backwards compatibility (used internally) +def build_prompt_from_checkpoint(checkpoint: CommitModel | None, recent_messages: list[TurnEvent], user_input: str) -> str: + checkpoints = [checkpoint] if checkpoint else [] + return build_prompt_from_checkpoints(checkpoints, recent_messages, user_input) + + +def _generate_commit_hash(repo_id: str, message: str) -> str: + content = {"repo_id": repo_id, "message": message, "ts": _utcnow().isoformat()} + return hashlib.sha256(json.dumps(content, sort_keys=True).encode()).hexdigest() + + +# ── Response/Request schemas ────────────────────────────────────────────────── + +class SessionResponse(BaseModel): + id: uuid.UUID + repo_id: Optional[uuid.UUID] + title: str + active_provider: str + active_model: str + seeded_commit_id: Optional[uuid.UUID] + forked_from_checkpoint_id: Optional[uuid.UUID] + branch_name: str + created_at: datetime + updated_at: datetime + + model_config = {"from_attributes": True} + + +class TurnResponse(BaseModel): + id: uuid.UUID + session_id: uuid.UUID + role: str + content: str + provider: str + model: str + sequence_number: int + created_at: datetime + + model_config = {"from_attributes": True} + + +class CreateSessionRequest(BaseModel): + repo_id: Optional[str] = None + title: str = "" + provider: str = "" + model: str = "" + seed_from: str = "head" # "head" | "none" | "" + + +class SendMessageRequest(BaseModel): + session_id: str + repo_id: Optional[str] = None + provider: str + model: str + message: str + use_mock: bool = Field( + False, + description="If true, use the deterministic mock adapter (no API key required)" + ) + memory_scope: str = Field( + "latest_1", + description="Memory scope for context: 'latest_1' or 'latest_3'" + ) + mounted_checkpoint_id: Optional[str] = Field( + None, + description="Explicit checkpoint id to anchor context. If set, overrides latest-head selection." + ) + history_base_seq: Optional[int] = Field( + None, + description="When mounting a specific commit, the sequence number of the last turn before mounting. Only turns with sequence_number > history_base_seq are included as history, preventing pre-mount turns from bleeding into the mounted context." + ) + + +class SendMessageResponse(BaseModel): + reply: str + session_id: uuid.UUID + turn_count: int + provider: str + model: str + + +class ManualCommitRequest(BaseModel): + repo_id: str + session_id: str + message: str + summary: str = "" + objective: str = "" + decisions: list[str] = Field(default_factory=list) + tasks: list[str] = Field(default_factory=list) + open_questions: list[str] = Field(default_factory=list) + entities: list[str] = Field(default_factory=list) + + +class CommitResponse(BaseModel): + id: uuid.UUID + repo_id: uuid.UUID + commit_hash: str + parent_commit_id: Optional[uuid.UUID] + branch_name: str + message: str + summary: str + objective: str + decisions: list + tasks: list + open_questions: list + entities: list + created_at: datetime + + model_config = {"from_attributes": True} + + +class HeadResponse(BaseModel): + repo_id: uuid.UUID + commit_hash: Optional[str] + commit_id: Optional[uuid.UUID] + summary: Optional[str] + objective: Optional[str] + latest_session_id: Optional[uuid.UUID] + latest_session_title: Optional[str] + + +# ── Session endpoints ───────────────────────────────────────────────────────── + +@router.post("/sessions", response_model=SessionResponse, status_code=201) +def create_session_generic(payload: CreateSessionRequest, db: Session = Depends(get_db)): + repo_id = uuid.UUID(payload.repo_id) if payload.repo_id else None + if repo_id: + _get_repo(repo_id, db) + + seeded_commit_id = None + if repo_id and payload.seed_from == "head": + latest = _get_latest_commit(repo_id, db) + if latest: + seeded_commit_id = latest.id + elif payload.seed_from not in ("none", "head", ""): + try: + c = db.get(CommitModel, uuid.UUID(payload.seed_from)) + if c and c.repo_id == repo_id: + seeded_commit_id = c.id + except (ValueError, Exception): + pass + + cfg = get_config() + provider = payload.provider or cfg.chat.default_provider + title = payload.title or f"Session {_utcnow().strftime('%b %d %H:%M')}" + + session = ChatSession( + repo_id=repo_id, + title=title, + active_provider=provider, + active_model=payload.model, + seeded_commit_id=seeded_commit_id, + ) + db.add(session) + db.commit() + db.refresh(session) + return session + + +@router.get("/sessions", response_model=list[SessionResponse]) +def list_recent_sessions(db: Session = Depends(get_db)): + stmt = select(ChatSession).order_by(ChatSession.updated_at.desc()).limit(50) + return db.scalars(stmt).all() + + +@router.get("/sessions/{session_id}", response_model=SessionResponse) +def get_session_generic(session_id: uuid.UUID, db: Session = Depends(get_db)): + session = db.get(ChatSession, session_id) + if not session: + raise HTTPException(status_code=404, detail="Session not found") + return session + + +@router.post("/sessions/{session_id}/title", response_model=SessionResponse) +def generate_session_title(session_id: uuid.UUID, db: Session = Depends(get_db)): + """Generate a meaningful title for a session using the background intelligence model.""" + session = db.get(ChatSession, session_id) + if not session: + raise HTTPException(status_code=404, detail="Session not found") + + turns = db.scalars( + select(TurnEvent) + .where(TurnEvent.session_id == session_id) + .order_by(TurnEvent.sequence_number) + .limit(4) + ).all() + if not turns: + raise HTTPException(status_code=400, detail="No turns to generate title from") + + transcript = "\n".join(f"{t.role.upper()}: {t.content[:300]}" for t in turns) + prompt = ( + "Generate a concise 3–5 word title for this conversation. " + "Output ONLY the title, no quotes, no punctuation, no explanation.\n\n" + f"{transcript}\n\nTitle:" + ) + + try: + cfg = get_config() + bg_provider = cfg.background.provider + bg_model = cfg.background.model + adapter = get_adapter(bg_provider, allow_mock=False) + raw_title = adapter.send([{"role": "user", "content": prompt}], model=bg_model).strip() + title = raw_title.strip("\"'").strip() + if len(title) > 60: + title = title[:60] + session.title = title + session.updated_at = _utcnow() + db.commit() + db.refresh(session) + except Exception: + pass # Return unchanged session if title generation fails + + return session + + +@router.get("/sessions/{session_id}/turns", response_model=list[TurnResponse]) +def list_turns_generic(session_id: uuid.UUID, db: Session = Depends(get_db)): + session = db.get(ChatSession, session_id) + if not session: + raise HTTPException(status_code=404, detail="Session not found") + stmt = select(TurnEvent).where(TurnEvent.session_id == session_id).order_by(TurnEvent.sequence_number) + return db.scalars(stmt).all() + + +class AttachSessionRequest(BaseModel): + repo_id: str + +@router.put("/sessions/{session_id}/attach", response_model=SessionResponse) +def attach_session(session_id: uuid.UUID, payload: AttachSessionRequest, db: Session = Depends(get_db)): + session = db.get(ChatSession, session_id) + if not session: + raise HTTPException(status_code=404, detail="Session not found") + repo_id = uuid.UUID(payload.repo_id) + _get_repo(repo_id, db) + + session.repo_id = repo_id + # Update turns to match new namespace + from sqlalchemy import update + db.execute(update(TurnEvent).where(TurnEvent.session_id == session_id).values(repo_id=repo_id)) + db.commit() + db.refresh(session) + return session + + +# Legacy detail routes for debug views +@router.post("/spaces/{repo_id}/sessions", response_model=SessionResponse, status_code=201) +def create_session( + repo_id: uuid.UUID, + payload: CreateSessionRequest, + db: Session = Depends(get_db), +): + payload.repo_id = str(repo_id) + return create_session_generic(payload, db) + +@router.get("/spaces/{repo_id}/sessions/{session_id}", response_model=SessionResponse) +def get_session( + repo_id: uuid.UUID, + session_id: uuid.UUID, + db: Session = Depends(get_db), +): + return get_session_generic(session_id, db) + +@router.get("/spaces/{repo_id}/sessions/{session_id}/turns", response_model=list[TurnResponse]) +def list_turns( + repo_id: uuid.UUID, + session_id: uuid.UUID, + db: Session = Depends(get_db), +): + return list_turns_generic(session_id, db) + + +# ── Send endpoint ───────────────────────────────────────────────────────────── + +@router.post("/send", response_model=SendMessageResponse) +def send_message(payload: SendMessageRequest, db: Session = Depends(get_db)): + """ + Send a user message and receive an assistant reply. + + Context strategy: + - On the FIRST user turn (sequence_number == 0): if the session has a + seeded_commit_id, inject a system message built from that commit snapshot. + - On subsequent turns: pass only the recent session turns as history. + - Provider switching: handled naturally — the new provider receives all + prior turns as conversation history (no extra re-injection). + """ + session_id = uuid.UUID(payload.session_id) + session = db.get(ChatSession, session_id) + if not session: + raise HTTPException(status_code=404, detail="Session not found") + + repo_id = session.repo_id + + if not payload.model: + raise HTTPException(status_code=400, detail="Model must be specified") + + # Determine next sequence number + stmt = ( + select(TurnEvent) + .where(TurnEvent.session_id == session_id) + .order_by(TurnEvent.sequence_number.desc()) + .limit(1) + ) + last_turn = db.scalars(stmt).first() + next_seq = (last_turn.sequence_number + 1) if last_turn else 0 + + # Determine effective checkpoint anchor. + # + # Priority order: + # 1. Explicit mounted_checkpoint_id from request (user is temporarily mounting a + # specific checkpoint in an existing session — the isolation boundary is + # history_base_seq supplied by the frontend at mount time). + # 2. session.forked_from_checkpoint_id (permanent fork — the session branched from + # this checkpoint and has no inherited live turns; history_base_seq is 0). + # 3. Scope-based HEAD resolution (normal HEAD mode). + effective_checkpoint_id = payload.mounted_checkpoint_id + effective_base_seq: Optional[int] = payload.history_base_seq + is_isolated = False + + if effective_checkpoint_id is not None and effective_base_seq is not None: + # Case 1: explicit temporary mount + is_isolated = True + elif effective_checkpoint_id is None and session.forked_from_checkpoint_id is not None: + # Case 2: forked session — auto-inherit, treat start-of-session as boundary + effective_checkpoint_id = str(session.forked_from_checkpoint_id) + effective_base_seq = 0 + is_isolated = True + + checkpoints = _resolve_checkpoints(repo_id, effective_checkpoint_id, payload.memory_scope, db) + latest_checkpoint = checkpoints[-1] if checkpoints else None + + # Get recent messages using the appropriate isolation boundary. + # + # Isolated mode (explicit mount OR forked session): only turns with + # sequence_number > effective_base_seq are included, so pre-fork / pre-mount + # turns from other branches cannot bleed in. + # + # HEAD mode: turns since the latest checkpoint was committed. + history_stmt = ( + select(TurnEvent) + .where(TurnEvent.session_id == session_id, TurnEvent.role != "system") + ) + if is_isolated: + history_stmt = history_stmt.where(TurnEvent.sequence_number > effective_base_seq) + elif latest_checkpoint: + # HEAD mode: turns since the checkpoint was created + history_stmt = history_stmt.where(TurnEvent.created_at >= latest_checkpoint.created_at) + + history_stmt = history_stmt.order_by(TurnEvent.sequence_number.asc()).limit(MAX_CONTEXT_TURNS) + recent_turns = db.scalars(history_stmt).all() + + # Reconstruct the unified prompt using Smriti memory engine (scope-aware) + prompt_text = build_prompt_from_checkpoints(checkpoints, recent_turns, payload.message) + + # Store user turn in DB (for future memory queries), but we send the reconstructed prompt to the LLM + user_turn = TurnEvent( + session_id=session_id, + repo_id=repo_id, + role="user", + content=payload.message, + provider=payload.provider, + model=payload.model, + sequence_number=next_seq, + ) + db.add(user_turn) + db.flush() + + # The reconstructed prompt bypasses standard chat roles to ensure strict cross-model continuation + messages = [{"role": "user", "content": prompt_text}] + + # Select adapter + try: + adapter = ( + get_mock_adapter() + if payload.use_mock + else get_adapter(payload.provider, allow_mock=False) + ) + except ProviderNotConfiguredError as e: + raise HTTPException(status_code=422, detail=str(e)) + + # Call provider + try: + reply_text = adapter.send(messages, model=payload.model) + except Exception as e: + raise HTTPException(status_code=502, detail=f"Provider error: {e}") + + # Store assistant turn + assistant_turn = TurnEvent( + session_id=session_id, + repo_id=repo_id, + role="assistant", + content=reply_text, + provider=payload.provider, + model=payload.model, + sequence_number=next_seq + 1, + ) + db.add(assistant_turn) + + # Update session active provider/model + session.active_provider = payload.provider + session.active_model = payload.model + session.updated_at = _utcnow() + + db.commit() + + turn_count = next_seq + 2 # user + assistant + return SendMessageResponse( + reply=reply_text, + session_id=session_id, + turn_count=turn_count, + provider=payload.provider, + model=payload.model, + ) + + +# ── Manual commit endpoint ──────────────────────────────────────────────────── + +@router.post("/commit", response_model=CommitResponse, status_code=201) +def manual_commit(payload: ManualCommitRequest, db: Session = Depends(get_db)): + """ + Manually create a Commit from the current session state. + The commit captures whatever structured state the user provides. + """ + repo_id = uuid.UUID(payload.repo_id) + session_id = uuid.UUID(payload.session_id) + + repo = _get_repo(repo_id, db) + session = db.get(ChatSession, session_id) + if not session or session.repo_id != repo_id: + raise HTTPException(status_code=404, detail="Session not found") + + # Derive branch identity from the session — the session row is the source of truth. + # For main-branch sessions: parent = latest main commit (existing behaviour). + # For fork-branch sessions: parent = latest commit on the session's own branch, + # falling back to the fork source checkpoint when no branch-local commits exist yet. + session_branch = session.branch_name # e.g. "main" or "branch-2026-03-21" + + if session_branch == "main": + parent = _get_latest_commit(repo_id, db) + else: + parent = _get_latest_commit_on_branch(repo_id, session_branch, db) + if parent is None and session.forked_from_checkpoint_id is not None: + # First checkpoint on this fork — its parent is the fork source + parent = db.get(CommitModel, session.forked_from_checkpoint_id) + + parent_id = parent.id if parent else None + + commit_hash = _generate_commit_hash(str(repo_id), payload.message) + + commit = CommitModel( + repo_id=repo_id, + commit_hash=commit_hash, + parent_commit_id=parent_id, + branch_name=session_branch, + author_agent=session.active_provider, + author_type="llm", + message=payload.message, + summary=payload.summary, + objective=payload.objective, + decisions=payload.decisions, + tasks=payload.tasks, + open_questions=payload.open_questions, + entities=payload.entities, + metadata_={"session_id": str(session_id)}, + ) + db.add(commit) + repo.updated_at = _utcnow() + db.commit() + db.refresh(commit) + return commit + + +# ── Head endpoint ───────────────────────────────────────────────────────────── + +@router.get("/spaces/{repo_id}/head", response_model=HeadResponse) +def get_head(repo_id: uuid.UUID, db: Session = Depends(get_db)): + """Return the latest commit + latest session metadata for a Space.""" + _get_repo(repo_id, db) + + latest_commit = _get_latest_commit(repo_id, db) + + # Latest session + session_stmt = ( + select(ChatSession) + .where(ChatSession.repo_id == repo_id) + .order_by(ChatSession.updated_at.desc()) + .limit(1) + ) + latest_session = db.scalars(session_stmt).first() + + return HeadResponse( + repo_id=repo_id, + commit_hash=latest_commit.commit_hash if latest_commit else None, + commit_id=latest_commit.id if latest_commit else None, + summary=latest_commit.summary if latest_commit else None, + objective=latest_commit.objective if latest_commit else None, + latest_session_id=latest_session.id if latest_session else None, + latest_session_title=latest_session.title if latest_session else None, + ) + + +# ── Provider status endpoint ────────────────────────────────────────────────── + +@router.get("/providers") +def list_providers(): + """Return provider configuration status. Safe to expose — never returns keys.""" + return providers_status() diff --git a/backend/app/api/routes/checkpoint.py b/backend/app/api/routes/checkpoint.py new file mode 100644 index 0000000..80415d0 --- /dev/null +++ b/backend/app/api/routes/checkpoint.py @@ -0,0 +1,156 @@ +"""Checkpoint routes for auto drafting.""" + +import json +import logging +import uuid +from typing import Optional + +from fastapi import APIRouter, Depends, HTTPException +from sqlalchemy import select +from sqlalchemy.orm import Session + +from app.db.database import get_db +from app.db.models import ChatSession, TurnEvent +from app.schemas import CheckpointDraftRequest, CheckpointDraftResponse +from app.providers.registry import get_adapter +from app.config_loader import get_config + +router = APIRouter() +logger = logging.getLogger(__name__) + + +def _fetch_turns_for_draft( + session_id: uuid.UUID, + mounted_checkpoint_id: Optional[str], + history_base_seq: Optional[int], + num_turns: int, + db: Session, +) -> list[TurnEvent]: + """ + Fetch the turns that should be included in the draft. + + Mirrors the three-way isolation logic used by send_message: + + Case 1 — explicit mount (mounted_checkpoint_id + history_base_seq both set): + Only turns with sequence_number > history_base_seq (post-mount turns only). + + Case 2 — forked session, no explicit mount: + The session row is the source of truth. If session.forked_from_checkpoint_id + is set and no explicit mount is active, apply sequence_number > 0 to include + only fork-local turns and exclude any hypothetical pre-fork leakage. + + Case 3 — main session, HEAD mode: + All turns in the session (no boundary applied here; the caller can + further restrict via num_turns). + + Capped at num_turns most recent turns, ordered oldest-first. + """ + # Load session to inspect fork identity — the session row is the source of truth. + session = db.get(ChatSession, session_id) + + stmt = ( + select(TurnEvent) + .where(TurnEvent.session_id == session_id, TurnEvent.role != "system") + ) + + if mounted_checkpoint_id is not None and history_base_seq is not None: + # Case 1: explicit temporary mount + stmt = stmt.where(TurnEvent.sequence_number > history_base_seq) + elif session is not None and session.forked_from_checkpoint_id is not None: + # Case 2: forked session, no explicit mount — only fork-local turns + stmt = stmt.where(TurnEvent.sequence_number > 0) + # Case 3: no filter — all session turns + + stmt = stmt.order_by(TurnEvent.sequence_number.desc()).limit(num_turns) + + # Reverse to chronological order for transcript building + return list(reversed(db.scalars(stmt).all())) + + +@router.post("/draft", response_model=CheckpointDraftResponse) +def draft_checkpoint(request: CheckpointDraftRequest, db: Session = Depends(get_db)): + # 1. Fetch Session + session = db.get(ChatSession, request.session_id) + if not session: + raise HTTPException(status_code=404, detail="Session not found") + + # 2. Fetch turns — respects mount isolation + turns = _fetch_turns_for_draft( + session_id=request.session_id, + mounted_checkpoint_id=request.mounted_checkpoint_id, + history_base_seq=request.history_base_seq, + num_turns=request.num_turns, + db=db, + ) + + if not turns: + return CheckpointDraftResponse() + + transcript = "" + for turn in turns: + transcript += f"{turn.role.upper()}: {turn.content}\n\n" + + # 3. Prompt — extract ONLY from the current conversation. + # No prior checkpoint context is injected to avoid cross-session contamination. + prompt = f"""You are a precise metadata extraction assistant. +Your task: extract structured information from the conversation below. +Extract ONLY what is explicitly discussed or decided in this conversation. +Do NOT infer, hallucinate, or carry over content from any other context. +If a field has nothing relevant in the conversation, return an empty string or empty array. + +CONVERSATION: +{transcript} + +Return a STRICT JSON object with exactly this schema — no extra keys, no markdown: +{{ + "title": "3-5 word title capturing the core topic of this conversation", + "objective": "The main goal the user is working toward in this conversation (1 sentence, or empty string if unclear)", + "summary": "Concise narrative of what was discussed and figured out (2-4 sentences)", + "decisions": ["An explicit decision made in the conversation", "Another explicit decision"], + "tasks": ["A concrete action item from the conversation", "Another action item"], + "open_questions": ["An unresolved question from the conversation"], + "entities": ["Key concept, tool, place, or system mentioned"] +}} + +Rules: +- decisions: only include choices explicitly made in the conversation, not hypothetical ones +- tasks: only include things the user said they will do or need to do +- entities: proper nouns and key technical/domain terms only +- All arrays may be empty if nothing relevant was discussed +- Output ONLY valid JSON. No markdown, no explanation. +""" + + messages = [{"role": "user", "content": prompt}] + + # 4. Call background intelligence provider + try: + cfg = get_config() + bg_provider = cfg.background.provider + bg_model = cfg.background.model + adapter = get_adapter(bg_provider, allow_mock=False) + except Exception as e: + raise HTTPException( + status_code=500, + detail=f"Background provider not configured in Settings. Error: {e}" + ) + + try: + raw_response = adapter.send(messages, model=bg_model, response_format={"type": "json_object"}) + data = json.loads(raw_response) + + return CheckpointDraftResponse( + title=str(data.get("title", "")).strip(), + objective=str(data.get("objective", "")).strip(), + summary=str(data.get("summary", "")).strip(), + decisions=list(dict.fromkeys([str(x).strip() for x in data.get("decisions", []) if x])), + tasks=list(dict.fromkeys([str(x).strip() for x in data.get("tasks", []) if x])), + open_questions=list(dict.fromkeys([str(x).strip() for x in data.get("open_questions", []) if x])), + entities=list(dict.fromkeys([str(x).strip() for x in data.get("entities", []) if x])), + ) + + except json.JSONDecodeError: + logger.error(f"LLM returned invalid JSON: {raw_response}") + raise HTTPException(status_code=502, detail="Failed to parse drafted checkpoint (invalid JSON from provider).") + except Exception as e: + logger.error(f"LLM extraction error: {e}") + raise HTTPException(status_code=502, detail=f"Drafting failed: {e}") diff --git a/backend/app/api/routes/commits.py b/backend/app/api/routes/commits.py new file mode 100644 index 0000000..05b3391 --- /dev/null +++ b/backend/app/api/routes/commits.py @@ -0,0 +1,96 @@ +import hashlib +import json +import uuid +from datetime import datetime + +from fastapi import APIRouter, Depends, HTTPException +from pydantic import BaseModel, Field +from sqlalchemy.orm import Session + +from app.db.database import get_db +from app.db.models import RepoModel, CommitModel +from app.api.routes.repos import CommitResponse, DEMO_USER_ID + +router = APIRouter(prefix="/commits", tags=["commits"]) + +class CommitCreate(BaseModel): + repo_id: str + parent_commit_id: str | None = None + branch_name: str = "main" + author_agent: str | None = None + author_type: str = Field("llm", description="user, llm, agent, system") + message: str + summary: str = "" + objective: str = "" + decisions: list = Field(default_factory=list) + tasks: list = Field(default_factory=list) + open_questions: list = Field(default_factory=list) + entities: list = Field(default_factory=list) + context_blob: dict = Field(default_factory=dict) + raw_source_text: str | None = None + metadata_: dict = Field(default_factory=dict, alias="metadata") + +def _generate_commit_hash(payload: CommitCreate) -> str: + """Generate a deterministic-ish commit hash based on state snapshot.""" + content = { + "repo_id": payload.repo_id, + "parent": payload.parent_commit_id, + "message": payload.message, + "summary": payload.summary, + "ts": datetime.utcnow().isoformat() + } + return hashlib.sha256(json.dumps(content, sort_keys=True).encode("utf-8")).hexdigest() + +@router.post("", response_model=CommitResponse, status_code=201) +def create_commit(payload: CommitCreate, db: Session = Depends(get_db)): + """Create a new commit (state snapshot).""" + repo = db.get(RepoModel, uuid.UUID(payload.repo_id)) + if not repo or repo.user_id != DEMO_USER_ID: + raise HTTPException(status_code=404, detail="Repo not found") + + parent_id = uuid.UUID(payload.parent_commit_id) if payload.parent_commit_id else None + if parent_id: + parent = db.get(CommitModel, parent_id) + if not parent or parent.repo_id != repo.id: + raise HTTPException(status_code=400, detail="Invalid parent commit") + + commit_hash = _generate_commit_hash(payload) + + new_commit = CommitModel( + repo_id=repo.id, + commit_hash=commit_hash, + parent_commit_id=parent_id, + branch_name=payload.branch_name, + author_agent=payload.author_agent, + author_type=payload.author_type, + message=payload.message, + summary=payload.summary, + objective=payload.objective, + decisions=payload.decisions, + tasks=payload.tasks, + open_questions=payload.open_questions, + entities=payload.entities, + context_blob=payload.context_blob, + raw_source_text=payload.raw_source_text, + metadata_=payload.metadata_ + ) + + db.add(new_commit) + repo.updated_at = datetime.utcnow() + db.commit() + db.refresh(new_commit) + + return new_commit + +@router.get("/{commit_id}", response_model=CommitResponse) +def get_commit(commit_id: uuid.UUID, db: Session = Depends(get_db)): + """Get a specific commit.""" + commit = db.get(CommitModel, commit_id) + if not commit: + raise HTTPException(status_code=404, detail="Commit not found") + + repo = db.get(RepoModel, commit.repo_id) + if not repo or repo.user_id != DEMO_USER_ID: + raise HTTPException(status_code=404, detail="Commit/Repo not found") + + return commit diff --git a/backend/app/api/routes/context_git.py b/backend/app/api/routes/context_git.py new file mode 100644 index 0000000..bd65e00 --- /dev/null +++ b/backend/app/api/routes/context_git.py @@ -0,0 +1,147 @@ +import uuid +from datetime import datetime + +from fastapi import APIRouter, Depends, HTTPException +from pydantic import BaseModel, Field +from sqlalchemy.orm import Session + +from app.db.database import get_db +from app.db.models import RepoModel, CommitModel + +router = APIRouter(prefix="/context", tags=["commits"]) + +# Assuming demo user id checking here if needed +DEMO_USER_ID = uuid.UUID("00000000-0000-0000-0000-00000000000a") + +class ContextFromCommitRequest(BaseModel): + commit_id: str + target: str = Field("generic", description="Target platform: chatgpt, claude, cursor, generic") + +class ContextBuildResponse(BaseModel): + content: str + target_tool: str + format: str = "markdown" + +def _format_list(items: list, title: str) -> str: + if not items: + return "" + bulleted = "\n".join(f"- {item}" for item in items) + return f"## {title}\n{bulleted}\n\n" + +@router.post("/from-commit", response_model=ContextBuildResponse) +def build_context_from_commit(payload: ContextFromCommitRequest, db: Session = Depends(get_db)): + """Generate a structured continuation context from a specific commit.""" + commit = db.get(CommitModel, uuid.UUID(payload.commit_id)) + if not commit: + raise HTTPException(status_code=404, detail="Commit not found") + + repo = db.get(RepoModel, commit.repo_id) + if not repo or repo.user_id != DEMO_USER_ID: + raise HTTPException(status_code=404, detail="Repo/Commit access denied") + + # Build structured text depending on target + content = "" + target = payload.target.lower() + + if target == "cursor": + content += f"# Project State: {repo.name}\n\n" + content += f"Commit Ref: `{commit.commit_hash}`\n" + content += f"Branch: `{commit.branch_name}`\n\n" + if commit.summary: + content += f"## Summary\n{commit.summary}\n\n" + if commit.objective: + content += f"## Current Objective\n{commit.objective}\n\n" + + content += _format_list(commit.tasks, "Active Tasks") + content += _format_list(commit.decisions, "Key Decisions") + content += _format_list(commit.open_questions, "Open Questions") + content += _format_list(commit.entities, "Entities") + + content += "## Instructions for Cursor\n" + content += "Please review the tasks and open questions above, and guide me on the next implementation steps." + + elif target == "claude": + content += "\n" + content += f" \n" + content += f" \n" + if commit.summary: + content += f" {commit.summary}\n" + if commit.objective: + content += f" {commit.objective}\n" + + if commit.decisions: + content += " \n" + "\n".join(f" {d}" for d in commit.decisions) + "\n \n" + if commit.tasks: + content += " \n" + "\n".join(f" {t}" for t in commit.tasks) + "\n \n" + content += "\n\n" + content += "Please assume this state and help me continue working toward the objective." + + else: + # Generic / ChatGPT + content += f"--- SMRITI CONTEXT PACK ---\n" + content += f"Repo: {repo.name}\n" + content += f"Commit: {commit.commit_hash}\n" + content += f"---\n\n" + if commit.summary: + content += f"**Summary**: {commit.summary}\n\n" + if commit.objective: + content += f"**Objective**: {commit.objective}\n\n" + + content += _format_list(commit.decisions, "Decisions") + content += _format_list(commit.tasks, "Tasks") + content += _format_list(commit.open_questions, "Open Questions") + content += _format_list(commit.entities, "Entities") + + content += "Please use this context as our shared starting point. Acknowledge and let's begin." + + return ContextBuildResponse( + content=content.strip(), + target_tool=payload.target, + ) + + +class CommitSnapshot(BaseModel): + """Minimal commit fields needed for delta comparison.""" + id: uuid.UUID + commit_hash: str + message: str + summary: str + objective: str + decisions: list + tasks: list + open_questions: list + entities: list + author_agent: str | None + author_type: str + branch_name: str + parent_commit_id: uuid.UUID | None + created_at: datetime + + model_config = {"from_attributes": True} + + +class ParentDeltaResponse(BaseModel): + current: CommitSnapshot + parent: CommitSnapshot | None + + +@router.get("/parent-delta/{commit_id}", response_model=ParentDeltaResponse) +def get_parent_delta(commit_id: uuid.UUID, db: Session = Depends(get_db)): + """Return the current commit and its parent (if any) so the frontend can compute diffs.""" + commit = db.get(CommitModel, commit_id) + if not commit: + raise HTTPException(status_code=404, detail="Commit not found") + + repo = db.get(RepoModel, commit.repo_id) + if not repo or repo.user_id != DEMO_USER_ID: + raise HTTPException(status_code=404, detail="Commit access denied") + + parent = None + if commit.parent_commit_id: + parent = db.get(CommitModel, commit.parent_commit_id) + + return ParentDeltaResponse( + current=CommitSnapshot.model_validate(commit), + parent=CommitSnapshot.model_validate(parent) if parent else None, + ) + diff --git a/backend/app/api/routes/lineage.py b/backend/app/api/routes/lineage.py new file mode 100644 index 0000000..76e251b --- /dev/null +++ b/backend/app/api/routes/lineage.py @@ -0,0 +1,412 @@ +""" +V5 Lineage API routes. + +Session forking: + POST /api/v5/lineage/sessions/fork – fork a new session from a checkpoint + +Branch/lineage view: + GET /api/v5/lineage/spaces/{space_id} – full checkpoint + session tree for a space + +Checkpoint comparison: + GET /api/v5/lineage/checkpoints/{a_id}/compare/{b_id} – structured diff of two checkpoints +""" + +from __future__ import annotations + +import uuid +from datetime import datetime, timezone +from typing import Optional + +from fastapi import APIRouter, Depends, HTTPException +from pydantic import BaseModel, Field +from sqlalchemy import select +from sqlalchemy.orm import Session + +from app.db.database import get_db +from app.db.models import ChatSession, CommitModel, RepoModel, TurnEvent +from app.config_loader import get_config + +router = APIRouter(prefix="/lineage", tags=["lineage-v5"]) + +DEMO_USER_ID = uuid.UUID("00000000-0000-0000-0000-00000000000a") + + +def _utcnow() -> datetime: + return datetime.now(timezone.utc) + + +def _get_repo(space_id: uuid.UUID, db: Session) -> RepoModel: + repo = db.get(RepoModel, space_id) + if not repo or repo.user_id != DEMO_USER_ID: + raise HTTPException(status_code=404, detail="Space not found") + return repo + + +def _get_commit(checkpoint_id: uuid.UUID, db: Session) -> CommitModel: + commit = db.get(CommitModel, checkpoint_id) + if not commit: + raise HTTPException(status_code=404, detail="Checkpoint not found") + return commit + + +# ── Request / Response schemas ──────────────────────────────────────────────── + +class ForkSessionRequest(BaseModel): + space_id: str + checkpoint_id: str + branch_name: str = "" + provider: str = "" + model: str = "" + + +class ForkSessionResponse(BaseModel): + session_id: uuid.UUID + branch_name: str + forked_from_checkpoint_id: uuid.UUID + # history_base_seq is always 0 for new forks: the session starts clean. + # This is an implementation detail exposed so the frontend can pass it + # back on the first send_message call if needed (though the backend also + # auto-infers it for forked sessions). + history_base_seq: int = 0 + + +class CheckpointNode(BaseModel): + id: uuid.UUID + commit_hash: str + message: str + branch_name: str + parent_checkpoint_id: Optional[uuid.UUID] + created_at: datetime + summary: str + objective: str + + model_config = {"from_attributes": True} + + +class SessionNode(BaseModel): + id: uuid.UUID + title: str + branch_name: str + forked_from_checkpoint_id: Optional[uuid.UUID] + seeded_commit_id: Optional[uuid.UUID] + created_at: datetime + + model_config = {"from_attributes": True} + + +class LineageResponse(BaseModel): + space_id: uuid.UUID + checkpoints: list[CheckpointNode] + sessions: list[SessionNode] + + +class CheckpointDetail(BaseModel): + id: uuid.UUID + commit_hash: str + message: str + branch_name: str + summary: str + objective: str + decisions: list + tasks: list + open_questions: list + + +class CheckpointDiff(BaseModel): + summary_a: str + summary_b: str + objective_a: str + objective_b: str + decisions_only_a: list[str] + decisions_only_b: list[str] + decisions_shared: list[str] + tasks_only_a: list[str] + tasks_only_b: list[str] + tasks_shared: list[str] + + +class CompareResponse(BaseModel): + checkpoint_a: CheckpointDetail + checkpoint_b: CheckpointDetail + diff: CheckpointDiff + + +class ReachableCheckpoint(BaseModel): + """Full commit data returned by the reachable-checkpoints endpoint. + Matches the Commit TypeScript type so the frontend can reuse existing types.""" + id: uuid.UUID + repo_id: uuid.UUID + commit_hash: str + parent_commit_id: Optional[uuid.UUID] + branch_name: str + author_agent: Optional[str] + author_type: str + message: str + summary: str + objective: str + decisions: list + tasks: list + open_questions: list + entities: list + context_blob: dict + raw_source_text: Optional[str] + metadata_: dict = Field(serialization_alias="metadata") + created_at: datetime + + model_config = {"from_attributes": True} + + +# ── Helpers ─────────────────────────────────────────────────────────────────── + +def _extract_text(item) -> str: + """Normalise a decision/task entry to plain text regardless of storage shape.""" + if isinstance(item, str): + return item + if isinstance(item, dict): + return item.get("description", item.get("text", str(item))) + return str(item) + + +def _diff_lists(a: list, b: list) -> tuple[list[str], list[str], list[str]]: + """Return (only_in_a, only_in_b, in_both) comparing by normalised text.""" + set_a = {_extract_text(x) for x in a} + set_b = {_extract_text(x) for x in b} + return ( + sorted(set_a - set_b), + sorted(set_b - set_a), + sorted(set_a & set_b), + ) + + +# ── Fork endpoint ───────────────────────────────────────────────────────────── + +@router.post("/sessions/fork", response_model=ForkSessionResponse, status_code=201) +def fork_session(payload: ForkSessionRequest, db: Session = Depends(get_db)): + """ + Create a new session branching from a specific checkpoint. + + The forked session: + - starts with no live turns (history_base_seq = 0) + - receives its context exclusively from the checkpoint state snapshot + - accumulates its own turns independently from that point forward + """ + try: + space_id = uuid.UUID(payload.space_id) + checkpoint_id = uuid.UUID(payload.checkpoint_id) + except ValueError: + raise HTTPException(status_code=400, detail="Invalid space_id or checkpoint_id") + + _get_repo(space_id, db) + commit = _get_commit(checkpoint_id, db) + + if commit.repo_id != space_id: + raise HTTPException(status_code=400, detail="Checkpoint does not belong to this space") + + cfg = get_config() + provider = payload.provider or cfg.chat.default_provider + branch = payload.branch_name or f"branch-{_utcnow().strftime('%Y-%m-%d')}" + + session = ChatSession( + repo_id=space_id, + title=f"Fork from {commit.commit_hash[:7]}", + active_provider=provider, + active_model=payload.model, + seeded_commit_id=checkpoint_id, + forked_from_checkpoint_id=checkpoint_id, + branch_name=branch, + ) + db.add(session) + db.commit() + db.refresh(session) + + return ForkSessionResponse( + session_id=session.id, + branch_name=session.branch_name, + forked_from_checkpoint_id=checkpoint_id, + history_base_seq=0, + ) + + +# ── Lineage (branch tree) endpoint ──────────────────────────────────────────── + +@router.get("/spaces/{space_id}", response_model=LineageResponse) +def get_lineage(space_id: uuid.UUID, db: Session = Depends(get_db)): + """ + Return all checkpoints and sessions for a space, structured for tree rendering. + + The frontend builds the visual branch tree from this data: + - checkpoints carry parent_checkpoint_id for the commit ancestry chain + - sessions carry forked_from_checkpoint_id to locate where each branch began + """ + _get_repo(space_id, db) + + commits = db.scalars( + select(CommitModel) + .where(CommitModel.repo_id == space_id) + .order_by(CommitModel.created_at.asc()) + ).all() + + sessions = db.scalars( + select(ChatSession) + .where(ChatSession.repo_id == space_id) + .order_by(ChatSession.created_at.asc()) + ).all() + + checkpoint_nodes = [ + CheckpointNode( + id=c.id, + commit_hash=c.commit_hash, + message=c.message, + branch_name=c.branch_name, + parent_checkpoint_id=c.parent_commit_id, + created_at=c.created_at, + summary=c.summary or "", + objective=c.objective or "", + ) + for c in commits + ] + + session_nodes = [ + SessionNode( + id=s.id, + title=s.title or "", + branch_name=s.branch_name, + forked_from_checkpoint_id=s.forked_from_checkpoint_id, + seeded_commit_id=s.seeded_commit_id, + created_at=s.created_at, + ) + for s in sessions + ] + + return LineageResponse( + space_id=space_id, + checkpoints=checkpoint_nodes, + sessions=session_nodes, + ) + + +# ── Compare endpoint ────────────────────────────────────────────────────────── + +@router.get("/checkpoints/{a_id}/compare/{b_id}", response_model=CompareResponse) +def compare_checkpoints(a_id: uuid.UUID, b_id: uuid.UUID, db: Session = Depends(get_db)): + """ + Return a structured diff of two checkpoint state snapshots. + + Decisions and tasks are compared by normalised text equality. + Summary and objective are returned as-is for side-by-side reading. + """ + commit_a = _get_commit(a_id, db) + commit_b = _get_commit(b_id, db) + + dec_only_a, dec_only_b, dec_shared = _diff_lists( + commit_a.decisions or [], commit_b.decisions or [] + ) + task_only_a, task_only_b, task_shared = _diff_lists( + commit_a.tasks or [], commit_b.tasks or [] + ) + + def _to_detail(c: CommitModel) -> CheckpointDetail: + return CheckpointDetail( + id=c.id, + commit_hash=c.commit_hash, + message=c.message, + branch_name=c.branch_name, + summary=c.summary or "", + objective=c.objective or "", + decisions=[_extract_text(d) for d in (c.decisions or [])], + tasks=[_extract_text(t) for t in (c.tasks or [])], + open_questions=[_extract_text(q) for q in (c.open_questions or [])], + ) + + return CompareResponse( + checkpoint_a=_to_detail(commit_a), + checkpoint_b=_to_detail(commit_b), + diff=CheckpointDiff( + summary_a=commit_a.summary or "", + summary_b=commit_b.summary or "", + objective_a=commit_a.objective or "", + objective_b=commit_b.objective or "", + decisions_only_a=dec_only_a, + decisions_only_b=dec_only_b, + decisions_shared=dec_shared, + tasks_only_a=task_only_a, + tasks_only_b=task_only_b, + tasks_shared=task_shared, + ), + ) + + +# ── Reachable checkpoints endpoint ──────────────────────────────────────────── + +@router.get("/sessions/{session_id}/checkpoints", response_model=list[ReachableCheckpoint]) +def get_session_reachable_checkpoints(session_id: uuid.UUID, db: Session = Depends(get_db)): + """ + Return the reachable checkpoint set for a session. + + Reachability rules (same-Space, branch-local semantics): + + Main-branch session: + - All commits where branch_name == 'main', newest first. + + Forked session (session.forked_from_checkpoint_id is set): + - Commits on the session's own branch (fork-local, created after fork). + - The fork source checkpoint itself. + - All ancestors of the fork source (walking parent_commit_id upward). + NOT included: + - Child commits on main created AFTER the fork point. + - Sibling branch commits. + + This is the single authoritative reachability query used by the + checkpoint history panel and mount-candidate list. It ensures that + the panel never offers cross-branch checkpoints as mount targets. + """ + session = db.get(ChatSession, session_id) + if not session: + raise HTTPException(status_code=404, detail="Session not found") + + if session.branch_name == "main" or session.forked_from_checkpoint_id is None: + # Main-branch session: all main commits, newest first + commits = db.scalars( + select(CommitModel) + .where( + CommitModel.repo_id == session.repo_id, + CommitModel.branch_name == "main", + ) + .order_by(CommitModel.created_at.desc()) + ).all() + return list(commits) + + # Forked session: collect fork-local checkpoints + ancestors of fork source + seen_ids: set[uuid.UUID] = set() + result: list[CommitModel] = [] + + # 1. Fork-local commits (same branch as the session, created after the fork) + fork_local = db.scalars( + select(CommitModel) + .where( + CommitModel.repo_id == session.repo_id, + CommitModel.branch_name == session.branch_name, + ) + .order_by(CommitModel.created_at.desc()) + ).all() + for c in fork_local: + if c.id not in seen_ids: + seen_ids.add(c.id) + result.append(c) + + # 2. Fork source + all its ancestors (walk upward through parent_commit_id) + # This includes the exact checkpoint the session was forked from and any + # earlier history — but NOT any commits created on main after that point. + current: Optional[CommitModel] = db.get(CommitModel, session.forked_from_checkpoint_id) + while current is not None: + if current.id not in seen_ids: + seen_ids.add(current.id) + result.append(current) + if current.parent_commit_id is None: + break + current = db.get(CommitModel, current.parent_commit_id) + + # Sort newest first (fork-local commits are already newest-first, but the + # ancestor walk may interleave with them if branches share commit timestamps) + result.sort(key=lambda c: c.created_at, reverse=True) + return result diff --git a/backend/app/api/routes/repos.py b/backend/app/api/routes/repos.py new file mode 100644 index 0000000..5d89554 --- /dev/null +++ b/backend/app/api/routes/repos.py @@ -0,0 +1,127 @@ +import uuid +from datetime import datetime +from typing import Annotated + +from typing import Optional + +from fastapi import APIRouter, Depends, HTTPException, Query +from pydantic import BaseModel, Field +from sqlalchemy import select +from sqlalchemy.orm import Session + +from app.db.database import get_db +from app.db.models import RepoModel, CommitModel + +router = APIRouter(prefix="/repos", tags=["repos"]) + +DEMO_USER_ID = uuid.UUID("00000000-0000-0000-0000-00000000000a") + +class RepoCreate(BaseModel): + name: str = Field(..., description="Name of the repo/project") + description: str = Field("", description="Optional description") + user_id: str | None = Field(None, description="Optional user ID, defaults to demo user") + metadata_: dict = Field(default_factory=dict, alias="metadata") + +class RepoResponse(BaseModel): + id: uuid.UUID + repo_slug: str | None + name: str + description: str + user_id: uuid.UUID + metadata_: dict = Field(serialization_alias="metadata") + created_at: datetime + updated_at: datetime + + model_config = {"from_attributes": True} + +class CommitResponse(BaseModel): + id: uuid.UUID + repo_id: uuid.UUID + commit_hash: str + parent_commit_id: uuid.UUID | None + branch_name: str + author_agent: str | None + author_type: str + message: str + summary: str + objective: str + decisions: list + tasks: list + open_questions: list + entities: list + context_blob: dict + raw_source_text: str | None + metadata_: dict = Field(serialization_alias="metadata") + created_at: datetime + + model_config = {"from_attributes": True} + +import logging +logger = logging.getLogger("uvicorn.error") + +@router.post("", response_model=RepoResponse, status_code=201) +def create_repo(payload: RepoCreate, db: Session = Depends(get_db)): + """Create a new memory repository.""" + logger.info("request received: POST /api/v2/repos") + new_repo = RepoModel( + user_id=uuid.UUID(payload.user_id) if payload.user_id else DEMO_USER_ID, + name=payload.name, + description=payload.description, + metadata_=payload.metadata_ + ) + logger.info("before DB call: add") + db.add(new_repo) + logger.info("before DB call: commit") + db.commit() + logger.info("before DB call: refresh") + db.refresh(new_repo) + logger.info("after DB call / before response return") + return new_repo + +@router.get("", response_model=list[RepoResponse]) +def list_repos(db: Session = Depends(get_db)): + """List all repos for the current user.""" + stmt = select(RepoModel).where(RepoModel.user_id == DEMO_USER_ID).order_by(RepoModel.updated_at.desc()) + return db.scalars(stmt).all() + +@router.get("/{repo_id}", response_model=RepoResponse) +def get_repo(repo_id: uuid.UUID, db: Session = Depends(get_db)): + """Get a specific repo.""" + repo = db.get(RepoModel, repo_id) + if not repo or repo.user_id != DEMO_USER_ID: + raise HTTPException(status_code=404, detail="Repo not found") + return repo + +@router.get("/{repo_id}/commits", response_model=list[CommitResponse]) +def list_repo_commits( + repo_id: uuid.UUID, + branch: Optional[str] = Query(None, description="Filter by branch name"), + db: Session = Depends(get_db), +): + """List commits for a repo. Optionally filter by branch name.""" + repo = db.get(RepoModel, repo_id) + if not repo or repo.user_id != DEMO_USER_ID: + raise HTTPException(status_code=404, detail="Repo not found") + + stmt = select(CommitModel).where(CommitModel.repo_id == repo_id) + if branch: + stmt = stmt.where(CommitModel.branch_name == branch) + stmt = stmt.order_by(CommitModel.created_at.desc()) + return db.scalars(stmt).all() + +@router.get("/{repo_id}/commits/latest", response_model=CommitResponse) +def get_latest_commit(repo_id: uuid.UUID, branch: str = "main", db: Session = Depends(get_db)): + """Get the latest commit for a repo (and optional branch).""" + repo = db.get(RepoModel, repo_id) + if not repo or repo.user_id != DEMO_USER_ID: + raise HTTPException(status_code=404, detail="Repo not found") + + stmt = select(CommitModel).where( + CommitModel.repo_id == repo_id, + CommitModel.branch_name == branch + ).order_by(CommitModel.created_at.desc()).limit(1) + + commit = db.scalars(stmt).first() + if not commit: + raise HTTPException(status_code=404, detail="No commits found for this repo/branch") + return commit diff --git a/backend/app/config.py b/backend/app/config.py new file mode 100644 index 0000000..06e8833 --- /dev/null +++ b/backend/app/config.py @@ -0,0 +1,24 @@ +from pydantic_settings import BaseSettings + + +class Settings(BaseSettings): + """Application settings loaded from environment variables.""" + + # App + app_name: str = "Smriti" + debug: bool = False + + # Database + database_url: str = "postgresql://smriti:smriti@localhost:5432/smriti" + + # OpenAI (for extraction service) + openai_api_key: str = "" + openai_model: str = "gpt-4o-mini" + + # CORS + cors_origins: list[str] = ["http://localhost:5173", "http://localhost:3000"] + + model_config = {"env_file": ".env", "env_file_encoding": "utf-8"} + + +settings = Settings() diff --git a/backend/app/config_loader.py b/backend/app/config_loader.py new file mode 100644 index 0000000..6dc2933 --- /dev/null +++ b/backend/app/config_loader.py @@ -0,0 +1,180 @@ +""" +Provider configuration loader. +Priority: env vars > config/providers.yaml > built-in defaults. +""" +from __future__ import annotations + +import os +from dataclasses import dataclass, field +from pathlib import Path + +try: + import yaml + _YAML_AVAILABLE = True +except ImportError: + _YAML_AVAILABLE = False + + +# ── Typed config objects ────────────────────────────────────────────────────── + +@dataclass +class ProviderConfig: + enabled: bool = False + api_key: str = "" + default_model: str = "" + base_url: str = "" + missing_package: bool = False + + +@dataclass +class ChatConfig: + default_provider: str = "openrouter" + autosave_turns: int = 50 + auto_commit: bool = False + +@dataclass +class BackgroundConfig: + provider: str = "openai" + model: str = "gpt-4o-mini" + + +@dataclass +class AppProviderConfig: + openai: ProviderConfig = field(default_factory=ProviderConfig) + anthropic: ProviderConfig = field(default_factory=ProviderConfig) + openrouter: ProviderConfig = field(default_factory=ProviderConfig) + chat: ChatConfig = field(default_factory=ChatConfig) + background: BackgroundConfig = field(default_factory=BackgroundConfig) + + +class ProviderNotConfiguredError(Exception): + """Raised when a requested provider is missing its API key.""" + + +# ── Config loader ───────────────────────────────────────────────────────────── + +_CONFIG_PATH = Path(__file__).parent.parent / "config" / "providers.yaml" + + +import logging +logger = logging.getLogger(__name__) + +def _load_yaml() -> dict: + if not _YAML_AVAILABLE: + logger.warning("PyYAML not installed, skipping providers.yaml config load") + return {} + if not _CONFIG_PATH.exists(): + logger.info(f"Config file not found: {_CONFIG_PATH} (using env vars / defaults)") + return {} + with _CONFIG_PATH.open("r") as f: + return yaml.safe_load(f) or {} + + +def load_config() -> AppProviderConfig: + raw = _load_yaml() + providers_raw = raw.get("providers", {}) + chat_raw = raw.get("chat", {}) + + def _provider(name: str, env_var: str, base_url_default: str = "") -> ProviderConfig: + """Merge YAML config + env override for a provider.""" + p = providers_raw.get(name, {}) + api_key = ( + os.environ.get(env_var) + or p.get("api_key", "") + ) + enabled = bool(api_key) and p.get("enabled", bool(api_key)) + return ProviderConfig( + enabled=enabled, + api_key=api_key, + default_model=p.get("default_model", ""), + base_url=p.get("base_url", base_url_default), + ) + + openai = _provider("openai", "OPENAI_API_KEY") + anthropic = _provider("anthropic", "ANTHROPIC_API_KEY") + openrouter = _provider( + "openrouter", "OPENROUTER_API_KEY", + base_url_default="https://openrouter.ai/api/v1", + ) + + chat = ChatConfig( + default_provider=os.environ.get("SMRITI_DEFAULT_PROVIDER") + or chat_raw.get("default_provider", "openrouter"), + autosave_turns=int(chat_raw.get("autosave_turns", 50)), + auto_commit=bool(chat_raw.get("auto_commit", False)), + ) + + bg_raw = raw.get("background_intelligence", {}) + background = BackgroundConfig( + provider=bg_raw.get("provider", "openai"), + model=bg_raw.get("model", "gpt-4o-mini") + ) + + return AppProviderConfig( + openai=openai, + anthropic=anthropic, + openrouter=openrouter, + chat=chat, + background=background, + ) + + +# Singleton — loaded once at import time +_config: AppProviderConfig | None = None + + +def get_config() -> AppProviderConfig: + global _config + if _config is None: + _config = load_config() + return _config + + +def get_provider_config(provider: str) -> ProviderConfig: + cfg = get_config() + pc = getattr(cfg, provider.lower(), None) + if pc is None: + raise ProviderNotConfiguredError(f"Unknown provider: {provider}") + if not pc.api_key: + raise ProviderNotConfiguredError( + f"Provider '{provider}' has no API key configured. " + f"Set the environment variable or add it to config/providers.yaml." + ) + return pc + + +def _check_package(provider: str) -> bool: + try: + if provider in ("openai", "openrouter"): + import openai # noqa + elif provider == "anthropic": + import anthropic # noqa + return False + except ImportError: + return True + + +def providers_status() -> dict[str, dict]: + """Return a dict summarising which providers are enabled — safe to expose in API.""" + cfg = get_config() + status = { + name: { + "enabled": getattr(cfg, name).enabled, + "has_key": bool(getattr(cfg, name).api_key), + "missing_package": _check_package(name), + "configured": getattr(cfg, name).enabled and bool(getattr(cfg, name).api_key) and not _check_package(name), + "status_label": "Ready" if getattr(cfg, name).enabled else "Disabled", + "default_model": getattr(cfg, name).default_model, + } + for name in ("openai", "anthropic", "openrouter") + } + status["background_intelligence"] = { + "provider": cfg.background.provider, + "model": cfg.background.model, + "enabled": True, + "has_key": True, + "missing_package": False, + "configured": True, + "status_label": "Ready", + } + return status diff --git a/backend/app/db/__init__.py b/backend/app/db/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/backend/app/db/database.py b/backend/app/db/database.py new file mode 100644 index 0000000..54ef541 --- /dev/null +++ b/backend/app/db/database.py @@ -0,0 +1,29 @@ +"""SQLAlchemy database engine and session management.""" + +from sqlalchemy import create_engine +from sqlalchemy.orm import DeclarativeBase, sessionmaker + +from app.config import settings + +engine = create_engine(settings.database_url, echo=settings.debug) +SessionLocal = sessionmaker(autocommit=False, autoflush=False, bind=engine) + + +class Base(DeclarativeBase): + pass + + +import logging +logger = logging.getLogger("uvicorn.error") + +def get_db(): + """Dependency that yields a database session.""" + logger.info("get_db: Creating new session...") + db = SessionLocal() + logger.info("get_db: Session created.") + try: + logger.info("get_db: Yielding session...") + yield db + finally: + logger.info("get_db: Closing session...") + db.close() diff --git a/backend/app/db/models.py b/backend/app/db/models.py new file mode 100644 index 0000000..13c63fc --- /dev/null +++ b/backend/app/db/models.py @@ -0,0 +1,254 @@ +"""SQLAlchemy ORM models for all database tables.""" + +import uuid +from datetime import datetime, timezone + +from sqlalchemy import DateTime, Float, ForeignKey, Integer, String, Text +from sqlalchemy.dialects.postgresql import JSONB, UUID +from sqlalchemy.orm import Mapped, mapped_column, relationship +from pgvector.sqlalchemy import Vector + +from app.db.database import Base + + +def _utcnow() -> datetime: + return datetime.now(timezone.utc) + + +class SessionModel(Base): + __tablename__ = "sessions" + + id: Mapped[uuid.UUID] = mapped_column( + UUID(as_uuid=True), primary_key=True, default=uuid.uuid4 + ) + title: Mapped[str | None] = mapped_column(String(255), nullable=True) + source_tool: Mapped[str | None] = mapped_column(String(50), nullable=True) + raw_transcript: Mapped[str] = mapped_column(Text, nullable=False) + status: Mapped[str] = mapped_column(String(20), default="processing") + created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=_utcnow) + updated_at: Mapped[datetime] = mapped_column( + DateTime(timezone=True), default=_utcnow, onupdate=_utcnow + ) + + # Relationships + messages: Mapped[list["MessageModel"]] = relationship( + back_populates="session", cascade="all, delete-orphan", order_by="MessageModel.position" + ) + extraction_result: Mapped["ExtractionResultModel | None"] = relationship( + back_populates="session", cascade="all, delete-orphan", uselist=False + ) + context_packs: Mapped[list["ContextPackModel"]] = relationship( + back_populates="session", cascade="all, delete-orphan" + ) + + +class MessageModel(Base): + __tablename__ = "messages" + + id: Mapped[uuid.UUID] = mapped_column( + UUID(as_uuid=True), primary_key=True, default=uuid.uuid4 + ) + session_id: Mapped[uuid.UUID] = mapped_column( + UUID(as_uuid=True), ForeignKey("sessions.id", ondelete="CASCADE"), nullable=False + ) + role: Mapped[str] = mapped_column(String(20), nullable=False) + content: Mapped[str] = mapped_column(Text, nullable=False) + position: Mapped[int] = mapped_column(Integer, nullable=False) + created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=_utcnow) + + # Relationships + session: Mapped["SessionModel"] = relationship(back_populates="messages") + + +class ExtractionResultModel(Base): + __tablename__ = "extraction_results" + + id: Mapped[uuid.UUID] = mapped_column( + UUID(as_uuid=True), primary_key=True, default=uuid.uuid4 + ) + session_id: Mapped[uuid.UUID] = mapped_column( + UUID(as_uuid=True), + ForeignKey("sessions.id", ondelete="CASCADE"), + nullable=False, + unique=True, + ) + summary: Mapped[str] = mapped_column(Text, nullable=False, default="") + decisions: Mapped[dict] = mapped_column(JSONB, default=list) + tasks: Mapped[dict] = mapped_column(JSONB, default=list) + open_questions: Mapped[dict] = mapped_column(JSONB, default=list) + entities: Mapped[dict] = mapped_column(JSONB, default=list) + code_snippets: Mapped[dict] = mapped_column(JSONB, default=list) + created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=_utcnow) + + # Relationships + session: Mapped["SessionModel"] = relationship(back_populates="extraction_result") + + +class ContextPackModel(Base): + __tablename__ = "context_packs" + + id: Mapped[uuid.UUID] = mapped_column( + UUID(as_uuid=True), primary_key=True, default=uuid.uuid4 + ) + session_id: Mapped[uuid.UUID] = mapped_column( + UUID(as_uuid=True), ForeignKey("sessions.id", ondelete="CASCADE"), nullable=False + ) + target_tool: Mapped[str] = mapped_column(String(20), nullable=False) + content: Mapped[str] = mapped_column(Text, nullable=False) + format: Mapped[str] = mapped_column(String(20), default="markdown") + created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=_utcnow) + + # Relationships + session: Mapped["SessionModel"] = relationship(back_populates="context_packs") + + +class MemoryItemModel(Base): + __tablename__ = "memories" + + id: Mapped[uuid.UUID] = mapped_column( + UUID(as_uuid=True), primary_key=True, default=uuid.uuid4 + ) + user_id: Mapped[uuid.UUID] = mapped_column( + UUID(as_uuid=True), nullable=False, index=True + ) + type: Mapped[str] = mapped_column(String(50), nullable=False) + content: Mapped[str] = mapped_column(Text, nullable=False) + embedding: Mapped[list[float] | None] = mapped_column(Vector(1536), nullable=True) # OpenAI small model dim + source: Mapped[str] = mapped_column(String(255), nullable=True) + confidence: Mapped[float] = mapped_column(Float, default=1.0) + importance: Mapped[float] = mapped_column(Float, default=1.0) + status: Mapped[str] = mapped_column(String(50), default="active") + metadata_: Mapped[dict] = mapped_column("metadata", JSONB, default=dict) + created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=_utcnow) + updated_at: Mapped[datetime] = mapped_column( + DateTime(timezone=True), default=_utcnow, onupdate=_utcnow + ) + +class RepoModel(Base): + __tablename__ = "repos" + + id: Mapped[uuid.UUID] = mapped_column( + UUID(as_uuid=True), primary_key=True, default=uuid.uuid4 + ) + repo_slug: Mapped[str | None] = mapped_column(String(255), unique=True, nullable=True) + name: Mapped[str] = mapped_column(String(255), nullable=False) + description: Mapped[str] = mapped_column(Text, default="") + user_id: Mapped[uuid.UUID] = mapped_column( + UUID(as_uuid=True), nullable=False, index=True + ) + metadata_: Mapped[dict] = mapped_column("metadata", JSONB, default=dict) + created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=_utcnow) + updated_at: Mapped[datetime] = mapped_column( + DateTime(timezone=True), default=_utcnow, onupdate=_utcnow + ) + + # Relationships + commits: Mapped[list["CommitModel"]] = relationship( + back_populates="repo", cascade="all, delete-orphan", order_by="CommitModel.created_at.desc()" + ) + +class CommitModel(Base): + __tablename__ = "commits" + + id: Mapped[uuid.UUID] = mapped_column( + UUID(as_uuid=True), primary_key=True, default=uuid.uuid4 + ) + repo_id: Mapped[uuid.UUID] = mapped_column( + UUID(as_uuid=True), ForeignKey("repos.id", ondelete="CASCADE"), nullable=False + ) + commit_hash: Mapped[str] = mapped_column(String(64), unique=True, nullable=False) + parent_commit_id: Mapped[uuid.UUID | None] = mapped_column( + UUID(as_uuid=True), ForeignKey("commits.id", ondelete="SET NULL"), nullable=True + ) + branch_name: Mapped[str] = mapped_column(String(255), default="main") + author_agent: Mapped[str | None] = mapped_column(String(255), nullable=True) + author_type: Mapped[str] = mapped_column(String(50), default="user") # user, llm, agent, system + message: Mapped[str] = mapped_column(String(255), nullable=False) + + # State snapshots + summary: Mapped[str] = mapped_column(Text, default="") + objective: Mapped[str] = mapped_column(Text, default="") + decisions: Mapped[dict] = mapped_column(JSONB, default=list) + tasks: Mapped[dict] = mapped_column(JSONB, default=list) + open_questions: Mapped[dict] = mapped_column(JSONB, default=list) + entities: Mapped[dict] = mapped_column(JSONB, default=list) + context_blob: Mapped[dict] = mapped_column(JSONB, default=dict) + + raw_source_text: Mapped[str | None] = mapped_column(Text, nullable=True) + metadata_: Mapped[dict] = mapped_column("metadata", JSONB, default=dict) + created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=_utcnow) + + # Relationships + repo: Mapped["RepoModel"] = relationship(back_populates="commits") + parent_commit: Mapped["CommitModel"] = relationship(remote_side=[id]) + + +# ── V4 Chat Models ──────────────────────────────────────────────────────────── + +class ChatSession(Base): + """A live conversation runtime inside a Space (Repo).""" + __tablename__ = "chat_sessions" + + id: Mapped[uuid.UUID] = mapped_column( + UUID(as_uuid=True), primary_key=True, default=uuid.uuid4 + ) + repo_id: Mapped[uuid.UUID | None] = mapped_column( + UUID(as_uuid=True), ForeignKey("repos.id", ondelete="CASCADE"), nullable=True, index=True + ) + title: Mapped[str] = mapped_column(String(255), default="") + active_provider: Mapped[str] = mapped_column(String(50), default="openrouter") + active_model: Mapped[str] = mapped_column(String(255), default="") + # The commit that was used to seed this session at open time + seeded_commit_id: Mapped[uuid.UUID | None] = mapped_column( + UUID(as_uuid=True), ForeignKey("commits.id", ondelete="SET NULL"), nullable=True + ) + # Forking: which checkpoint this session branched from (None = not a fork) + forked_from_checkpoint_id: Mapped[uuid.UUID | None] = mapped_column( + UUID(as_uuid=True), ForeignKey("commits.id", ondelete="SET NULL"), nullable=True + ) + # Branch identity — "main" for primary sessions, custom name for forks + branch_name: Mapped[str] = mapped_column(String(255), default="main") + metadata_: Mapped[dict] = mapped_column("metadata", JSONB, default=dict) + created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=_utcnow) + updated_at: Mapped[datetime] = mapped_column( + DateTime(timezone=True), default=_utcnow, onupdate=_utcnow + ) + + # Relationships + turns: Mapped[list["TurnEvent"]] = relationship( + back_populates="session", + cascade="all, delete-orphan", + order_by="TurnEvent.sequence_number", + ) + + +class TurnEvent(Base): + """One user or assistant message turn inside a ChatSession.""" + __tablename__ = "turn_events" + + id: Mapped[uuid.UUID] = mapped_column( + UUID(as_uuid=True), primary_key=True, default=uuid.uuid4 + ) + session_id: Mapped[uuid.UUID] = mapped_column( + UUID(as_uuid=True), ForeignKey("chat_sessions.id", ondelete="CASCADE"), + nullable=False, index=True + ) + repo_id: Mapped[uuid.UUID | None] = mapped_column( + UUID(as_uuid=True), ForeignKey("repos.id", ondelete="CASCADE"), + nullable=True, index=True + ) + role: Mapped[str] = mapped_column(String(20), nullable=False) # user | assistant | system + content: Mapped[str] = mapped_column(Text, nullable=False) + provider: Mapped[str] = mapped_column(String(50), default="") + model: Mapped[str] = mapped_column(String(255), default="") + sequence_number: Mapped[int] = mapped_column(Integer, nullable=False, default=0) + # Optional back-link to a commit that summarises this turn range + commit_id: Mapped[uuid.UUID | None] = mapped_column( + UUID(as_uuid=True), ForeignKey("commits.id", ondelete="SET NULL"), nullable=True + ) + metadata_: Mapped[dict] = mapped_column("metadata", JSONB, default=dict) + created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=_utcnow) + + # Relationships + session: Mapped["ChatSession"] = relationship(back_populates="turns") + diff --git a/backend/app/domain/__init__.py b/backend/app/domain/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/backend/app/domain/enums.py b/backend/app/domain/enums.py new file mode 100644 index 0000000..872f627 --- /dev/null +++ b/backend/app/domain/enums.py @@ -0,0 +1,31 @@ +from enum import StrEnum + + +class TargetTool(StrEnum): + """Supported target tools for context pack generation.""" + CHATGPT = "chatgpt" + CLAUDE = "claude" + CURSOR = "cursor" + GENERIC = "generic" + + +class SourceTool(StrEnum): + """Source tools that generated the original transcript.""" + CHATGPT = "chatgpt" + CLAUDE = "claude" + CURSOR = "cursor" + OTHER = "other" + + +class SessionStatus(StrEnum): + """Processing status of a session.""" + PROCESSING = "processing" + COMPLETED = "completed" + FAILED = "failed" + + +class MessageRole(StrEnum): + """Role of a message sender in a transcript.""" + HUMAN = "human" + ASSISTANT = "assistant" + UNKNOWN = "unknown" diff --git a/backend/app/domain/models.py b/backend/app/domain/models.py new file mode 100644 index 0000000..3171779 --- /dev/null +++ b/backend/app/domain/models.py @@ -0,0 +1,78 @@ +"""Domain models — pure data structures, no IO.""" + +from dataclasses import dataclass, field + +from app.domain.enums import MessageRole, SessionStatus, TargetTool + + +@dataclass +class Message: + """A single message parsed from a transcript.""" + role: MessageRole + content: str + position: int + + +@dataclass +class Decision: + """A key decision extracted from a session.""" + description: str + context: str = "" + + +@dataclass +class Task: + """A task extracted from a session.""" + description: str + status: str = "pending" + + +@dataclass +class OpenQuestion: + """An unresolved question from a session.""" + question: str + context: str = "" + + +@dataclass +class Entity: + """An important entity mentioned in a session.""" + name: str + type: str # e.g., "project", "file", "technology", "person" + context: str = "" + + +@dataclass +class CodeSnippet: + """A significant code snippet from a session.""" + language: str + code: str + description: str = "" + + +@dataclass +class ExtractionResult: + """The full set of artifacts extracted from a session.""" + summary: str + decisions: list[Decision] = field(default_factory=list) + tasks: list[Task] = field(default_factory=list) + open_questions: list[OpenQuestion] = field(default_factory=list) + entities: list[Entity] = field(default_factory=list) + code_snippets: list[CodeSnippet] = field(default_factory=list) + + +@dataclass +class ExtractedMemory: + """A single discrete memory extracted from a conversation.""" + type: str # e.g., 'episodic', 'semantic', 'task', 'preference', 'decision' + content: str + confidence: float = 1.0 + importance: float = 1.0 + + +@dataclass +class ContextPack: + """A target-specific continuation pack.""" + target_tool: TargetTool + content: str + format: str = "markdown" diff --git a/backend/app/main.py b/backend/app/main.py new file mode 100644 index 0000000..47cc945 --- /dev/null +++ b/backend/app/main.py @@ -0,0 +1,75 @@ +import logging +import re +from fastapi import FastAPI +from fastapi.middleware.cors import CORSMiddleware + +from app.config import settings + +class SecretGuardFilter(logging.Filter): + def filter(self, record): + if isinstance(record.msg, str) and re.search(r'sk-[a-zA-Z0-9_\-]+', record.msg): + record.msg = re.sub(r'sk-[a-zA-Z0-9_\-]+', '[REDACTED_SECRET]', record.msg) + return True + + +def create_app() -> FastAPI: + """Create and configure the FastAPI application.""" + app = FastAPI( + title=settings.app_name, + description="Cross-agent memory handoff system", + version="0.1.0", + ) + + # CORS + app.add_middleware( + CORSMiddleware, + allow_origins=settings.cors_origins, + allow_credentials=True, + allow_methods=["*"], + allow_headers=["*"], + ) + + # Secret Guard + for handler in logging.root.handlers: + handler.addFilter(SecretGuardFilter()) + + from app.api.routes import repos, commits, context_git, chat, checkpoint, lineage + + # V2 Routes (Git for memory) + app.include_router(repos.router, prefix="/api/v2", tags=["repos"]) + app.include_router(commits.router, prefix="/api/v2", tags=["commits"]) + app.include_router(context_git.router, prefix="/api/v2", tags=["commits"]) + + # V4 Routes (Chat workspace) + app.include_router(chat.router, prefix="/api/v4", tags=["chat-v4"]) + + # V5 Routes + app.include_router(checkpoint.router, prefix="/api/v5/checkpoint", tags=["checkpoint-v5"]) + app.include_router(lineage.router, prefix="/api/v5", tags=["lineage-v5"]) + + @app.get("/health") + async def health_check(): + return {"status": "ok"} + + @app.on_event("startup") + async def startup_event(): + from app.config_loader import providers_status + import logging + logger = logging.getLogger("smriti.startup") + + status = providers_status() + logger.info("Provider runtime validation:") + for name, info in status.items(): + if info.get("missing_package", False): + logger.warning(f" - {name}: disabled (package not installed)") + elif not info.get("has_key", False): + logger.info(f" - {name}: disabled (no API key configured)") + elif not info.get("enabled", False): + logger.info(f" - {name}: disabled (explicitly disabled in config)") + else: + logger.info(f" - {name}: configured and ready") + + return app + + +app = create_app() diff --git a/backend/app/providers/__init__.py b/backend/app/providers/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/backend/app/providers/anthropic_adapter.py b/backend/app/providers/anthropic_adapter.py new file mode 100644 index 0000000..76ba568 --- /dev/null +++ b/backend/app/providers/anthropic_adapter.py @@ -0,0 +1,34 @@ +"""Anthropic provider adapter.""" +from __future__ import annotations + +import anthropic +from app.providers.base import ProviderAdapter + + +class AnthropicAdapter(ProviderAdapter): + def __init__(self, api_key: str): + self._client = anthropic.Anthropic(api_key=api_key) + + def send(self, messages: list[dict[str, str]], model: str, **kwargs) -> str: + # Anthropic's API separates system messages from the conversation turns + system_parts = [m["content"] for m in messages if m["role"] == "system"] + chat_messages = [m for m in messages if m["role"] != "system"] + + system_text = "\n".join(system_parts) + + resp = self._client.messages.create( + model=model, + max_tokens=kwargs.pop("max_tokens", 4096), + system=system_text or anthropic.NOT_GIVEN, + messages=chat_messages, # type: ignore[arg-type] + **kwargs, + ) + return resp.content[0].text if resp.content else "" + + def healthcheck(self) -> bool: + try: + # Lightweight call to validate credentials + self._client.models.list() + return True + except Exception: + return False diff --git a/backend/app/providers/base.py b/backend/app/providers/base.py new file mode 100644 index 0000000..c187e41 --- /dev/null +++ b/backend/app/providers/base.py @@ -0,0 +1,26 @@ +"""Abstract base for all provider adapters.""" +from __future__ import annotations + +from abc import ABC, abstractmethod + + +class ProviderAdapter(ABC): + """Normalised interface every provider adapter must implement.""" + + @abstractmethod + def send( + self, + messages: list[dict[str, str]], + model: str, + **kwargs, + ) -> str: + """ + Send a list of chat messages and return the assistant reply as a string. + + messages: [{"role": "user"|"assistant"|"system", "content": str}, ...] + model: provider-specific model slug + """ + + @abstractmethod + def healthcheck(self) -> bool: + """Return True if the provider is reachable and configured.""" diff --git a/backend/app/providers/openai_adapter.py b/backend/app/providers/openai_adapter.py new file mode 100644 index 0000000..99d2bb6 --- /dev/null +++ b/backend/app/providers/openai_adapter.py @@ -0,0 +1,28 @@ +"""OpenAI provider adapter.""" +from __future__ import annotations + +from app.providers.base import ProviderAdapter + + +class OpenAIAdapter(ProviderAdapter): + def __init__(self, api_key: str, base_url: str | None = None): + try: + from openai import OpenAI + except ImportError as e: + raise ImportError("Install 'openai' package: pip install openai") from e + self._client = OpenAI(api_key=api_key, base_url=base_url or None) + + def send(self, messages: list[dict[str, str]], model: str, **kwargs) -> str: + resp = self._client.chat.completions.create( + model=model, + messages=messages, # type: ignore[arg-type] + **kwargs, + ) + return resp.choices[0].message.content or "" + + def healthcheck(self) -> bool: + try: + self._client.models.list() + return True + except Exception: + return False diff --git a/backend/app/providers/openrouter_adapter.py b/backend/app/providers/openrouter_adapter.py new file mode 100644 index 0000000..7a6f38c --- /dev/null +++ b/backend/app/providers/openrouter_adapter.py @@ -0,0 +1,19 @@ +"""OpenRouter provider adapter. + +Uses the openai SDK pointed at OpenRouter's OpenAI-compatible API endpoint. +This gives access to 200+ models through a single key. +""" +from __future__ import annotations + +from app.providers.openai_adapter import OpenAIAdapter + +_DEFAULT_BASE_URL = "https://openrouter.ai/api/v1" + + +class OpenRouterAdapter(OpenAIAdapter): + def __init__(self, api_key: str, base_url: str = _DEFAULT_BASE_URL): + super().__init__(api_key=api_key, base_url=base_url) + + def healthcheck(self) -> bool: + # OpenRouter doesn't support models.list(), so just return True if the key exists + return bool(True) diff --git a/backend/app/providers/registry.py b/backend/app/providers/registry.py new file mode 100644 index 0000000..0378500 --- /dev/null +++ b/backend/app/providers/registry.py @@ -0,0 +1,68 @@ +"""Provider registry — resolves provider name → adapter instance.""" +from __future__ import annotations + +from app.config_loader import get_provider_config, ProviderNotConfiguredError +from app.providers.base import ProviderAdapter + + +class MockAdapter(ProviderAdapter): + """Deterministic mock adapter used in tests and when no real keys are present.""" + + def send(self, messages: list[dict[str, str]], model: str, **kwargs) -> str: + last_user = next( + (m["content"] for m in reversed(messages) if m["role"] == "user"), + "Hello", + ) + return ( + f"[mock:{model}] You said: \"{last_user[:80]}\". " + "This is a deterministic mock response from Smriti's provider layer. " + "Configure a real provider key to use a live model." + ) + + def healthcheck(self) -> bool: + return True + + +def get_adapter(provider: str, allow_mock: bool = False) -> ProviderAdapter: + """ + Return the adapter for the given provider name. + + If allow_mock=True and the provider has no key, returns MockAdapter instead + of raising an error. Useful for demos and tests. + """ + p = provider.lower() + + try: + cfg = get_provider_config(p) + except ProviderNotConfiguredError: + if allow_mock: + return MockAdapter() + raise + + if p == "openai": + try: + from app.providers.openai_adapter import OpenAIAdapter + return OpenAIAdapter(api_key=cfg.api_key) + except ImportError as e: + raise ProviderNotConfiguredError(str(e)) + + if p == "anthropic": + try: + from app.providers.anthropic_adapter import AnthropicAdapter + return AnthropicAdapter(api_key=cfg.api_key) + except ImportError as e: + raise ProviderNotConfiguredError(str(e)) + + if p == "openrouter": + try: + from app.providers.openrouter_adapter import OpenRouterAdapter + return OpenRouterAdapter(api_key=cfg.api_key, base_url=cfg.base_url or "https://openrouter.ai/api/v1") + except ImportError as e: + raise ProviderNotConfiguredError(str(e)) + + raise ProviderNotConfiguredError(f"Unknown provider: {provider}") + + +def get_mock_adapter() -> MockAdapter: + """Always return a mock adapter, for tests and offline demos.""" + return MockAdapter() diff --git a/backend/app/repositories/__init__.py b/backend/app/repositories/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/backend/app/repositories/context_pack_repo.py b/backend/app/repositories/context_pack_repo.py new file mode 100644 index 0000000..da14c0e --- /dev/null +++ b/backend/app/repositories/context_pack_repo.py @@ -0,0 +1,36 @@ +"""Repository for ContextPack persistence.""" + +import uuid + +from sqlalchemy.orm import Session + +from app.db.models import ContextPackModel + + +class ContextPackRepository: + def __init__(self, db: Session): + self.db = db + + def create_pack( + self, session_id: uuid.UUID, target_tool: str, content: str, format: str = "markdown" + ) -> ContextPackModel: + pack = ContextPackModel( + session_id=session_id, + target_tool=target_tool, + content=content, + format=format, + ) + self.db.add(pack) + self.db.commit() + self.db.refresh(pack) + return pack + + def get_pack(self, pack_id: uuid.UUID) -> ContextPackModel | None: + return self.db.query(ContextPackModel).filter(ContextPackModel.id == pack_id).first() + + def get_packs_for_session(self, session_id: uuid.UUID) -> list[ContextPackModel]: + return ( + self.db.query(ContextPackModel) + .filter(ContextPackModel.session_id == session_id) + .all() + ) diff --git a/backend/app/repositories/session_repo.py b/backend/app/repositories/session_repo.py new file mode 100644 index 0000000..e691cdd --- /dev/null +++ b/backend/app/repositories/session_repo.py @@ -0,0 +1,67 @@ +"""Repository for Session, Message, and ExtractionResult persistence.""" + +import uuid + +from sqlalchemy.orm import Session + +from app.db.models import ExtractionResultModel, MessageModel, SessionModel +from app.domain.enums import SessionStatus + + +class SessionRepository: + def __init__(self, db: Session): + self.db = db + + def create_session( + self, + raw_transcript: str, + title: str | None = None, + source_tool: str | None = None, + ) -> SessionModel: + session = SessionModel( + raw_transcript=raw_transcript, + title=title, + source_tool=source_tool, + status=SessionStatus.PROCESSING, + ) + self.db.add(session) + self.db.commit() + self.db.refresh(session) + return session + + def get_session(self, session_id: uuid.UUID) -> SessionModel | None: + return self.db.query(SessionModel).filter(SessionModel.id == session_id).first() + + def update_status(self, session_id: uuid.UUID, status: SessionStatus) -> None: + self.db.query(SessionModel).filter(SessionModel.id == session_id).update( + {"status": status} + ) + self.db.commit() + + def create_messages( + self, session_id: uuid.UUID, messages: list[dict] + ) -> list[MessageModel]: + msg_models = [ + MessageModel(session_id=session_id, **msg) for msg in messages + ] + self.db.add_all(msg_models) + self.db.commit() + return msg_models + + def create_extraction_result( + self, session_id: uuid.UUID, extraction_data: dict + ) -> ExtractionResultModel: + result = ExtractionResultModel(session_id=session_id, **extraction_data) + self.db.add(result) + self.db.commit() + self.db.refresh(result) + return result + + def get_extraction_result( + self, session_id: uuid.UUID + ) -> ExtractionResultModel | None: + return ( + self.db.query(ExtractionResultModel) + .filter(ExtractionResultModel.session_id == session_id) + .first() + ) diff --git a/backend/app/schemas/__init__.py b/backend/app/schemas/__init__.py new file mode 100644 index 0000000..dbe637d --- /dev/null +++ b/backend/app/schemas/__init__.py @@ -0,0 +1,115 @@ +"""API request and response schemas using Pydantic.""" + +import uuid +from datetime import datetime +from typing import Optional + +from pydantic import BaseModel, Field + +from app.domain.enums import SessionStatus, TargetTool + + +# ── Request Schemas ────────────────────────────────────────────────────────── + + +class SessionCreateRequest(BaseModel): + """Request body for creating a new session.""" + raw_transcript: str = Field(..., min_length=1, description="The raw transcript text") + title: str | None = Field(None, max_length=255, description="Optional session title") + source_tool: str | None = Field(None, description="Source AI tool") + + +class ContextPackCreateRequest(BaseModel): + """Request body for generating a context pack.""" + target_tool: TargetTool = Field(..., description="Target tool for continuation pack") + + +# ── Response Schemas ───────────────────────────────────────────────────────── + + +class SessionResponse(BaseModel): + """Response for a session.""" + id: uuid.UUID + title: str | None + source_tool: str | None + status: SessionStatus + raw_transcript: str + created_at: datetime + + model_config = {"from_attributes": True} + + +class DecisionResponse(BaseModel): + description: str + context: str = "" + + +class TaskResponse(BaseModel): + description: str + status: str = "pending" + + +class OpenQuestionResponse(BaseModel): + question: str + context: str = "" + + +class EntityResponse(BaseModel): + name: str + type: str + context: str = "" + + +class CodeSnippetResponse(BaseModel): + language: str + code: str + description: str = "" + + +class ArtifactsResponse(BaseModel): + """Response for extracted artifacts.""" + summary: str + decisions: list[DecisionResponse] + tasks: list[TaskResponse] + open_questions: list[OpenQuestionResponse] + entities: list[EntityResponse] + code_snippets: list[CodeSnippetResponse] + + +class ContextPackResponse(BaseModel): + """Response for a generated context pack.""" + id: uuid.UUID + session_id: uuid.UUID + target_tool: str + content: str + format: str + created_at: datetime + + model_config = {"from_attributes": True} + + +class ErrorResponse(BaseModel): + """Standard error response.""" + error: str + detail: str | None = None + +class CheckpointDraftRequest(BaseModel): + session_id: uuid.UUID + num_turns: int = Field(15, ge=1, le=100) + mounted_checkpoint_id: Optional[str] = Field( + None, + description="If set, draft only uses turns after history_base_seq (mirrors send_message isolation)." + ) + history_base_seq: Optional[int] = Field( + None, + description="Sequence boundary from mount event. Only turns with sequence_number > this value are included." + ) + +class CheckpointDraftResponse(BaseModel): + title: str = "" + objective: str = "" + summary: str = "" + decisions: list[str] = Field(default_factory=list) + tasks: list[str] = Field(default_factory=list) + open_questions: list[str] = Field(default_factory=list) + entities: list[str] = Field(default_factory=list) diff --git a/backend/app/services/__init__.py b/backend/app/services/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/backend/app/services/embedding.py b/backend/app/services/embedding.py new file mode 100644 index 0000000..fd54dc5 --- /dev/null +++ b/backend/app/services/embedding.py @@ -0,0 +1,47 @@ +"""Embedding service for semantic vectors.""" + +import random +from typing import Protocol + +from openai import AsyncOpenAI + + +class EmbeddingService(Protocol): + """Protocol for embedding generation.""" + + async def generate_embedding(self, text: str) -> list[float]: + """Generate a vector embedding for the given text.""" + ... + + +class OpenAIEmbeddingService: + """Uses OpenAI's text-embedding-3-small to generate embeddings.""" + + def __init__(self, api_key: str): + self.client = AsyncOpenAI(api_key=api_key) + + async def generate_embedding(self, text: str) -> list[float]: + if not text or not text.strip(): + return [0.0] * 1536 + + # Standard OpenAI embedding dimension for text-embedding-3-small is 1536 + response = await self.client.embeddings.create( + input=text, + model="text-embedding-3-small", + ) + return response.data[0].embedding + + +class MockEmbeddingService: + """Mock service for testing/local dev without an API key.""" + + async def generate_embedding(self, text: str) -> list[float]: + # Return a normalized random vector of size 1536 + if not text or not text.strip(): + return [0.0] * 1536 + + vec = [random.uniform(-1.0, 1.0) for _ in range(1536)] + magnitude = sum(x * x for x in vec) ** 0.5 + if magnitude == 0: + return vec + return [x / magnitude for x in vec] diff --git a/backend/app/services/extractor.py b/backend/app/services/extractor.py new file mode 100644 index 0000000..f3f3ebb --- /dev/null +++ b/backend/app/services/extractor.py @@ -0,0 +1,39 @@ +"""Extraction service — orchestrates LLM-based context extraction.""" + +from app.domain.models import ExtractionResult, Message +from app.services.llm.base import LLMProvider + + +class ExtractorService: + """Extracts structured artifacts from parsed messages using an LLM provider. + + The LLM provider is injected, enabling: + - MockProvider for tests (deterministic, no API calls) + - OpenAIProvider for production + """ + + def __init__(self, llm_provider: LLMProvider): + self.llm_provider = llm_provider + + async def extract(self, messages: list[Message]) -> ExtractionResult: + """Extract structured artifacts from parsed messages. + + Args: + messages: List of parsed Message objects from a transcript. + + Returns: + ExtractionResult containing summary, decisions, tasks, etc. + """ + result = await self.llm_provider.extract(messages) + return result + + async def extract_memories(self, messages: list[Message]) -> list['app.domain.models.ExtractedMemory']: + """Extract generic memory items from parsed messages. + + Args: + messages: List of parsed Message objects. + + Returns: + List of ExtractedMemory objects. + """ + return await self.llm_provider.extract_memories(messages) diff --git a/backend/app/services/llm/__init__.py b/backend/app/services/llm/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/backend/app/services/llm/base.py b/backend/app/services/llm/base.py new file mode 100644 index 0000000..f33ba59 --- /dev/null +++ b/backend/app/services/llm/base.py @@ -0,0 +1,31 @@ +"""LLM provider interface and base classes.""" + +from typing import Protocol + +from app.domain.models import ExtractionResult, Message + + +class LLMProvider(Protocol): + """Interface for LLM providers used in extraction.""" + + async def extract(self, messages: list[Message]) -> ExtractionResult: + """Extract structured artifacts from a list of messages. + + Args: + messages: Parsed messages from a transcript. + + Returns: + ExtractionResult with summary, decisions, tasks, etc. + """ + ... + + async def extract_memories(self, messages: list[Message]) -> list['app.domain.models.ExtractedMemory']: + """Extract generic memory items from a list of messages. + + Args: + messages: Parsed messages from a transcript. + + Returns: + List of generic extracted memories. + """ + ... diff --git a/backend/app/services/llm/mock_provider.py b/backend/app/services/llm/mock_provider.py new file mode 100644 index 0000000..ca15f0c --- /dev/null +++ b/backend/app/services/llm/mock_provider.py @@ -0,0 +1,111 @@ +"""Mock LLM provider for tests — returns deterministic extraction results.""" + +from app.domain.models import ( + CodeSnippet, + Decision, + Entity, + ExtractionResult, + Message, + OpenQuestion, + Task, +) + + +class MockProvider: + """Deterministic mock LLM provider for testing. + + Returns predictable extraction results based on input message content, + enabling tests to validate schema structure and pipeline behavior + without actual LLM calls. + """ + + async def extract(self, messages: list[Message]) -> ExtractionResult: + """Generate a deterministic extraction result from messages.""" + if not messages: + return ExtractionResult( + summary="No messages provided.", + decisions=[], + tasks=[], + open_questions=[], + entities=[], + code_snippets=[], + ) + + # Build a simple summary from messages + all_content = " ".join(m.content for m in messages) + word_count = len(all_content.split()) + summary = f"Session with {len(messages)} messages and approximately {word_count} words." + + # Extract simple decisions (lines starting with "Decision:" or containing "decided") + decisions = [] + tasks = [] + open_questions = [] + entities = [] + code_snippets = [] + + for msg in messages: + content = msg.content + + # Look for decision-like patterns + if "decided" in content.lower() or "decision" in content.lower(): + decisions.append(Decision( + description=f"Decision from message {msg.position}", + context=content[:200], + )) + + # Look for task-like patterns + if "todo" in content.lower() or "task" in content.lower() or "need to" in content.lower(): + tasks.append(Task( + description=f"Task from message {msg.position}", + status="pending", + )) + + # Look for question patterns + if "?" in content: + open_questions.append(OpenQuestion( + question=f"Question from message {msg.position}", + context=content[:200], + )) + + # Look for code blocks + if "```" in content: + code_snippets.append(CodeSnippet( + language="unknown", + code=content, + description=f"Code from message {msg.position}", + )) + + # Always ensure at least one entity + entities.append(Entity( + name="session", + type="concept", + context="The overall session", + )) + + return ExtractionResult( + summary=summary, + decisions=decisions, + tasks=tasks, + open_questions=open_questions, + entities=entities, + code_snippets=code_snippets, + ) + + async def extract_memories(self, messages: list[Message]) -> list['app.domain.models.ExtractedMemory']: + import app.domain.models as models + if not messages: + return [] + + memories = [] + for msg in messages: + content = msg.content + if "like" in content.lower() or "prefer" in content.lower(): + memories.append(models.ExtractedMemory(type="preference", content=content)) + elif "task" in content.lower() or "todo" in content.lower(): + memories.append(models.ExtractedMemory(type="task", content=content)) + elif "is a " in content.lower() or "uses " in content.lower(): + memories.append(models.ExtractedMemory(type="semantic", content=content)) + elif len(memories) < 2: + memories.append(models.ExtractedMemory(type="episodic", content=f"Discussed: {content[:50]}")) + + return memories diff --git a/backend/app/services/llm/openai_provider.py b/backend/app/services/llm/openai_provider.py new file mode 100644 index 0000000..5be49ab --- /dev/null +++ b/backend/app/services/llm/openai_provider.py @@ -0,0 +1,131 @@ +"""OpenAI LLM provider for production extraction.""" + +import json + +from openai import AsyncOpenAI + +from app.config import settings +from app.domain.models import ( + CodeSnippet, + Decision, + Entity, + ExtractionResult, + Message, + OpenQuestion, + Task, +) + +EXTRACTION_SYSTEM_PROMPT = """You are a precise context extraction engine. Given a conversation transcript, extract structured information. + +Return a JSON object with exactly these fields: +{ + "summary": "A concise 3-5 sentence summary of what was discussed and accomplished", + "decisions": [{"description": "what was decided", "context": "why or relevant context"}], + "tasks": [{"description": "what needs to be done", "status": "pending|in_progress|completed"}], + "open_questions": [{"question": "the question", "context": "relevant context"}], + "entities": [{"name": "entity name", "type": "project|file|technology|person|concept", "context": "how it was mentioned"}], + "code_snippets": [{"language": "the language", "code": "the code", "description": "what this code does"}] +} + +Rules: +- Be concise and actionable +- Only include genuinely important items +- Each decision should be a clear statement +- Each task should be concrete +- Entities should be unique and meaningful +- Code snippets should only include significant code, not trivial examples +- Return valid JSON only, no markdown formatting""" + + +MEMORY_EXTRACTION_SYSTEM_PROMPT = """You are a precise context extraction engine. Given a conversation transcript, extract discrete, structured memories. + +Return a JSON object with exactly this field: +{ + "memories": [ + { + "type": "episodic|semantic|task|preference|decision", + "content": "A clear, standalone statement of fact, decision, preference, past event, or pending task.", + "confidence": 0.9, + "importance": 0.8 + } + ] +} + +Rules: +- Be concise and actionable. +- Ensure each memory is entirely self-contained (do not just say "the user", use explicit names if possible, else "the user"). +- Limit output to genuinely important or useful information. +- Output ONLY valid JSON.""" + + +class OpenAIProvider: + """Production LLM provider using OpenAI API.""" + + def __init__(self): + self.client = AsyncOpenAI(api_key=settings.openai_api_key) + self.model = settings.openai_model + + async def extract(self, messages: list[Message]) -> ExtractionResult: + """Extract structured artifacts using OpenAI API.""" + if not messages: + return ExtractionResult(summary="No messages provided.") + + # Format messages for the prompt + formatted = "\n\n".join( + f"[{m.role.value.upper()}]: {m.content}" for m in messages + ) + + response = await self.client.chat.completions.create( + model=self.model, + messages=[ + {"role": "system", "content": EXTRACTION_SYSTEM_PROMPT}, + {"role": "user", "content": f"Extract context from this transcript:\n\n{formatted}"}, + ], + response_format={"type": "json_object"}, + temperature=0.1, + ) + + raw_json = response.choices[0].message.content + data = json.loads(raw_json) + + return ExtractionResult( + summary=data.get("summary", ""), + decisions=[Decision(**d) for d in data.get("decisions", [])], + tasks=[Task(**t) for t in data.get("tasks", [])], + open_questions=[OpenQuestion(**q) for q in data.get("open_questions", [])], + entities=[Entity(**e) for e in data.get("entities", [])], + code_snippets=[CodeSnippet(**s) for s in data.get("code_snippets", [])], + ) + + async def extract_memories(self, messages: list[Message]) -> list['app.domain.models.ExtractedMemory']: + import app.domain.models as models + if not messages: + return [] + + formatted = "\n\n".join( + f"[{m.role.value.upper()}]: {m.content}" for m in messages + ) + + response = await self.client.chat.completions.create( + model=self.model, + messages=[ + {"role": "system", "content": MEMORY_EXTRACTION_SYSTEM_PROMPT}, + {"role": "user", "content": f"Extract memories from this transcript:\n\n{formatted}"}, + ], + response_format={"type": "json_object"}, + temperature=0.1, + ) + + raw_json = response.choices[0].message.content + data = json.loads(raw_json) + + memories = [] + for item in data.get("memories", []): + memories.append(models.ExtractedMemory( + type=item.get("type", "semantic"), + content=item.get("content", ""), + confidence=float(item.get("confidence", 1.0)), + importance=float(item.get("importance", 1.0)), + )) + + return memories diff --git a/backend/app/services/pack_generator.py b/backend/app/services/pack_generator.py new file mode 100644 index 0000000..4ac6885 --- /dev/null +++ b/backend/app/services/pack_generator.py @@ -0,0 +1,347 @@ +"""Context pack generation service — deterministic template-based rendering.""" + +from app.domain.enums import TargetTool +from app.domain.models import ContextPack, ExtractionResult + + +def generate_pack(result: ExtractionResult, target: TargetTool) -> ContextPack: + """Generate a target-specific continuation pack from extraction results. + + This is a pure function — deterministic, no LLM, no IO. + Each target gets a format optimized for that tool's strengths. + + Args: + result: The extraction result containing all artifacts. + target: The target tool for the continuation pack. + + Returns: + ContextPack with rendered content. + """ + generators = { + TargetTool.CHATGPT: _generate_chatgpt_pack, + TargetTool.CLAUDE: _generate_claude_pack, + TargetTool.CURSOR: _generate_cursor_pack, + TargetTool.GENERIC: _generate_generic_pack, + } + + generator = generators[target] + content = generator(result) + + return ContextPack( + target_tool=target, + content=content, + format="markdown", + ) + + +def _generate_chatgpt_pack(result: ExtractionResult) -> str: + """Generate a conversational continuation prompt for ChatGPT.""" + lines = [] + lines.append("I'm continuing a previous work session. Here's the context you need to help me pick up where I left off:") + lines.append("") + + lines.append(f"**What we were working on:** {result.summary}") + lines.append("") + + if result.decisions: + lines.append("**Key decisions already made:**") + for d in result.decisions: + lines.append(f"- {d.description}") + lines.append("") + + if result.tasks: + pending = [t for t in result.tasks if t.status != "completed"] + if pending: + lines.append("**Outstanding tasks:**") + for t in pending: + lines.append(f"- {t.description} (status: {t.status})") + lines.append("") + + if result.open_questions: + lines.append("**Open questions to address:**") + for q in result.open_questions: + lines.append(f"- {q.question}") + lines.append("") + + if result.entities: + lines.append("**Important context:**") + entity_names = [e.name for e in result.entities] + lines.append(f"Key entities/concepts: {', '.join(entity_names)}") + lines.append("") + + if result.code_snippets: + lines.append("**Relevant code from the session:**") + for s in result.code_snippets: + lines.append(f"```{s.language}") + lines.append(s.code) + lines.append("```") + if s.description: + lines.append(f"({s.description})") + lines.append("") + + lines.append("Please help me continue this work. Start by confirming you understand the context, then let's proceed with the outstanding tasks.") + + return "\n".join(lines) + + +def _generate_claude_pack(result: ExtractionResult) -> str: + """Generate a structured continuation prompt for Claude with XML-style sections.""" + lines = [] + lines.append("I'm continuing work from a previous session. Here's the structured context:") + lines.append("") + + lines.append("") + lines.append(f"{result.summary}") + lines.append("") + + if result.decisions: + lines.append("") + for d in result.decisions: + lines.append(f" {d.description}") + lines.append("") + lines.append("") + + if result.tasks: + lines.append("") + for t in result.tasks: + lines.append(f" {t.description}") + lines.append("") + lines.append("") + + if result.open_questions: + lines.append("") + for q in result.open_questions: + lines.append(f" {q.question}") + lines.append("") + lines.append("") + + if result.entities: + lines.append("") + for e in result.entities: + lines.append(f" {e.name}") + lines.append("") + lines.append("") + + if result.code_snippets: + lines.append("") + for s in result.code_snippets: + lines.append(f" ") + lines.append(f" {s.code}") + lines.append(f" ") + lines.append("") + lines.append("") + + lines.append("") + lines.append("") + lines.append("Please review this context and help me continue. Focus on the outstanding tasks and open questions.") + + return "\n".join(lines) + + +def _generate_cursor_pack(result: ExtractionResult) -> str: + """Generate a developer-focused continuation pack for Cursor.""" + lines = [] + lines.append("# Continuation Context") + lines.append("") + lines.append(f"## Summary") + lines.append(result.summary) + lines.append("") + + if result.tasks: + lines.append("## Task Checklist") + for t in result.tasks: + checkbox = "x" if t.status == "completed" else " " + lines.append(f"- [{checkbox}] {t.description}") + lines.append("") + + if result.decisions: + lines.append("## Decisions") + for d in result.decisions: + lines.append(f"- {d.description}") + lines.append("") + + if result.entities: + files = [e for e in result.entities if e.type == "file"] + tech = [e for e in result.entities if e.type == "technology"] + other = [e for e in result.entities if e.type not in ("file", "technology")] + if files: + lines.append("## Relevant Files") + for e in files: + lines.append(f"- `{e.name}`") + lines.append("") + if tech: + lines.append("## Technologies") + for e in tech: + lines.append(f"- {e.name}") + lines.append("") + if other: + lines.append("## Key Entities") + for e in other: + lines.append(f"- {e.name} ({e.type})") + lines.append("") + + if result.code_snippets: + lines.append("## Code Context") + for s in result.code_snippets: + if s.description: + lines.append(f"### {s.description}") + lines.append(f"```{s.language}") + lines.append(s.code) + lines.append("```") + lines.append("") + + if result.open_questions: + lines.append("## Open Questions") + for q in result.open_questions: + lines.append(f"- {q.question}") + lines.append("") + + return "\n".join(lines) + + +def _generate_generic_pack(result: ExtractionResult) -> str: + """Generate a portable Markdown continuation pack.""" + lines = [] + lines.append("# Session Continuation Pack") + lines.append("") + lines.append("## Summary") + lines.append(result.summary) + lines.append("") + + if result.decisions: + lines.append("## Key Decisions") + for d in result.decisions: + lines.append(f"- **{d.description}**") + if d.context: + lines.append(f" Context: {d.context}") + lines.append("") + + if result.tasks: + lines.append("## Tasks") + for t in result.tasks: + status_icon = "✅" if t.status == "completed" else "⬜" + lines.append(f"- {status_icon} {t.description} [{t.status}]") + lines.append("") + + if result.open_questions: + lines.append("## Open Questions") + for q in result.open_questions: + lines.append(f"- {q.question}") + if q.context: + lines.append(f" Context: {q.context}") + lines.append("") + + if result.entities: + lines.append("## Entities") + for e in result.entities: + lines.append(f"- **{e.name}** ({e.type})") + lines.append("") + + if result.code_snippets: + lines.append("## Code Snippets") + for s in result.code_snippets: + if s.description: + lines.append(f"### {s.description}") + lines.append(f"```{s.language}") + lines.append(s.code) + lines.append("```") + lines.append("") + + return "\n".join(lines) + + +from typing import Any + +def generate_from_memories(memories: list[Any], target: TargetTool) -> ContextPack: + """Generate a target-specific continuation pack from a list of memory items. + + Unlike generate_pack which uses fixed ExtractionResult schemas, this + dynamically groups and renders any memory items. + """ + generators = { + TargetTool.CHATGPT: _generate_chatgpt_memories_pack, + TargetTool.CLAUDE: _generate_claude_memories_pack, + TargetTool.CURSOR: _generate_cursor_memories_pack, + TargetTool.GENERIC: _generate_generic_memories_pack, + } + + generator = generators[target] + content = generator(memories) + + return ContextPack( + target_tool=target, + content=content, + format="markdown", + ) + + +def _render_memories_markdown(memories: list[Any]) -> str: + """Helper to render a list of memories into structured markdown.""" + lines = [] + from collections import defaultdict + grouped = defaultdict(list) + for m in memories: + # handle dicts or objects + m_type = m["type"] if isinstance(m, dict) else m.type + grouped[m_type].append(m) + + order = ["summary", "decision", "preference", "episodic", "semantic", "task", "code"] + + for t in order: + if t in grouped: + lines.append(f"## {t.title()}s") + for m in grouped[t]: + m_content = m["content"] if isinstance(m, dict) else m.content + lines.append(f"- {m_content}") + lines.append("") + + for t, items in grouped.items(): + if t not in order: + lines.append(f"## {t.title()}s") + for m in items: + m_content = m["content"] if isinstance(m, dict) else m.content + lines.append(f"- {m_content}") + lines.append("") + + return "\n".join(lines) + + +def _generate_chatgpt_memories_pack(memories: list[Any]) -> str: + lines = [ + "I'm continuing a previous work session. Here is our shared memory context:", + "", + _render_memories_markdown(memories), + "Please help me continue this work. Start by confirming you understand the context." + ] + return "\n".join(lines) + + +def _generate_claude_memories_pack(memories: list[Any]) -> str: + lines = [ + "I'm continuing work from a previous session. Here is our shared memory context:", + "", + "", + _render_memories_markdown(memories), + "", + "", + "Please review this context and help me continue." + ] + return "\n".join(lines) + + +def _generate_cursor_memories_pack(memories: list[Any]) -> str: + lines = [ + "# Shared Memory Context", + "", + _render_memories_markdown(memories) + ] + return "\n".join(lines) + + +def _generate_generic_memories_pack(memories: list[Any]) -> str: + lines = [ + "# Session Continuation Pack", + "", + _render_memories_markdown(memories) + ] + return "\n".join(lines) diff --git a/backend/app/services/parser.py b/backend/app/services/parser.py new file mode 100644 index 0000000..76c2ee5 --- /dev/null +++ b/backend/app/services/parser.py @@ -0,0 +1,202 @@ +"""Transcript parsing service — pure function, no IO. + +Supports multiple real-world transcript formats: +- Standard: Human:/Assistant:, User:/AI:, You:/Bot:, ChatGPT:/Claude: +- ChatGPT web: "You said:" / "ChatGPT said:" +- ChatGPT shared links: "User" / "Assistant" on own line +- Markdown bold: **User**: / **Assistant**: +- Plain text fallback (single UNKNOWN message) + +Strategy: try each format detector in priority order; first match wins. +""" + +import re +from typing import Callable + +from app.domain.enums import MessageRole +from app.domain.models import Message + +# ── Human / Assistant keyword sets ─────────────────────────────────────── + +HUMAN_KEYWORDS = {"human", "user", "you"} +ASSISTANT_KEYWORDS = { + "assistant", "ai", "chatgpt", "claude", "cursor", "bot", + "gpt-4", "gpt-4o", "gpt", "copilot", +} + + +def _detect_role(marker: str) -> MessageRole: + """Map a role marker string to a MessageRole.""" + cleaned = marker.lower().strip().rstrip(":").strip("* ") + if cleaned in HUMAN_KEYWORDS: + return MessageRole.HUMAN + if cleaned in ASSISTANT_KEYWORDS: + return MessageRole.ASSISTANT + return MessageRole.UNKNOWN + + +# ═════════════════════════════════════════════════════════════════════════ +# Strategy 1: Standard "Role: content" on same line +# Matches: Human: ..., Assistant: ..., User: ..., AI: ..., ChatGPT: ... +# ═════════════════════════════════════════════════════════════════════════ + +_STANDARD_ROLE_WORDS = "|".join(HUMAN_KEYWORDS | ASSISTANT_KEYWORDS) + +# Pattern matches "Role:" at start of line (case-insensitive) +_STANDARD_SPLIT = re.compile( + rf"^({_STANDARD_ROLE_WORDS}):\s*", + re.IGNORECASE | re.MULTILINE, +) + + +def _try_standard(raw: str) -> list[Message] | None: + parts = _STANDARD_SPLIT.split(raw) + if len(parts) <= 1: + return None + return _parts_to_messages(parts) + + +# ═════════════════════════════════════════════════════════════════════════ +# Strategy 2: ChatGPT web copy-paste — "You said:" / "ChatGPT said:" +# ═════════════════════════════════════════════════════════════════════════ + +_CHATGPT_WEB_SPLIT = re.compile( + r"^(You|ChatGPT|User|Assistant)\s+said:\s*", + re.IGNORECASE | re.MULTILINE, +) + + +def _try_chatgpt_web(raw: str) -> list[Message] | None: + parts = _CHATGPT_WEB_SPLIT.split(raw) + if len(parts) <= 1: + return None + return _parts_to_messages(parts) + + +# ═════════════════════════════════════════════════════════════════════════ +# Strategy 3: ChatGPT shared link / "Role" on own line (no colon) +# Matches: A line that is ONLY "User" or "Assistant" (possibly with +# markdown heading markers), followed by content on next lines. +# ═════════════════════════════════════════════════════════════════════════ + +_SHARED_LINK_SPLIT = re.compile( + rf"^#{{0,3}}\s*({_STANDARD_ROLE_WORDS})\s*$", + re.IGNORECASE | re.MULTILINE, +) + + +def _try_shared_link(raw: str) -> list[Message] | None: + parts = _SHARED_LINK_SPLIT.split(raw) + if len(parts) <= 1: + return None + return _parts_to_messages(parts) + + +# ═════════════════════════════════════════════════════════════════════════ +# Strategy 4: Markdown bold — **User**: / **Assistant**: +# ═════════════════════════════════════════════════════════════════════════ + +_MARKDOWN_BOLD_SPLIT = re.compile( + rf"^\*\*({_STANDARD_ROLE_WORDS})\*\*:\s*", + re.IGNORECASE | re.MULTILINE, +) + + +def _try_markdown_bold(raw: str) -> list[Message] | None: + parts = _MARKDOWN_BOLD_SPLIT.split(raw) + if len(parts) <= 1: + return None + return _parts_to_messages(parts) + + +# ═════════════════════════════════════════════════════════════════════════ +# Strategy 5: Angle-bracket / HTML-style — / +# ═════════════════════════════════════════════════════════════════════════ + +_ANGLE_BRACKET_SPLIT = re.compile( + rf"^<({_STANDARD_ROLE_WORDS})>\s*", + re.IGNORECASE | re.MULTILINE, +) + + +def _try_angle_bracket(raw: str) -> list[Message] | None: + parts = _ANGLE_BRACKET_SPLIT.split(raw) + if len(parts) <= 1: + return None + return _parts_to_messages(parts) + + +# ═════════════════════════════════════════════════════════════════════════ +# Shared helper: convert regex split parts into Message list +# ═════════════════════════════════════════════════════════════════════════ + +def _parts_to_messages(parts: list[str]) -> list[Message] | None: + """Convert regex split output (pre-text, marker, content, ...) to Messages. + + `re.split()` with a capturing group produces: + [pre_text, marker1, content1, marker2, content2, ...] + """ + messages: list[Message] = [] + position = 0 + + # Start at index 1 to skip pre-marker text + i = 1 + while i < len(parts) - 1: + marker = parts[i] + content = parts[i + 1].strip() + if content: + role = _detect_role(marker) + messages.append(Message(role=role, content=content, position=position)) + position += 1 + i += 2 + + return messages if messages else None + + +# ═════════════════════════════════════════════════════════════════════════ +# Main entry point +# ═════════════════════════════════════════════════════════════════════════ + +# Strategies in priority order — first successful match wins +_STRATEGIES: list[Callable[[str], list[Message] | None]] = [ + _try_standard, + _try_chatgpt_web, + _try_markdown_bold, + _try_angle_bracket, + _try_shared_link, # Last among structured — it's greedy (bare "User" on a line) +] + + +def parse_transcript(raw: str) -> list[Message]: + """Parse a raw transcript into a list of Message objects. + + Tries multiple format detection strategies in order: + 1. Standard role labels (Human:/Assistant:, User:/AI:, etc.) + 2. ChatGPT web copy-paste ("You said:" / "ChatGPT said:") + 3. Markdown bold (**User**: / **Assistant**:) + 4. Angle-bracket ( / ) + 5. Shared-link style (bare "User" / "Assistant" on own line) + 6. Fallback: single UNKNOWN message + + All strategies: + - Preserve code blocks intact within messages + - Handle multi-line message content + - Assign sequential position numbers + + Args: + raw: The raw transcript text. + + Returns: + List of Message objects. Empty list for empty/whitespace input. + """ + if not raw or not raw.strip(): + return [] + + # Try each strategy + for strategy in _STRATEGIES: + result = strategy(raw) + if result: + return result + + # Fallback: entire transcript as single unknown message + return [Message(role=MessageRole.UNKNOWN, content=raw.strip(), position=0)] diff --git a/backend/config/.gitignore b/backend/config/.gitignore new file mode 100644 index 0000000..934f0ac --- /dev/null +++ b/backend/config/.gitignore @@ -0,0 +1 @@ +providers.yaml diff --git a/backend/config/providers.example.yaml b/backend/config/providers.example.yaml new file mode 100644 index 0000000..9ebaf05 --- /dev/null +++ b/backend/config/providers.example.yaml @@ -0,0 +1,35 @@ +# Smriti provider configuration +# +# Copy this file to providers.yaml and fill in your API keys. +# providers.yaml is gitignored and will never be committed. +# +# Priority: environment variables > providers.yaml > defaults +# All api_key values can also be set via environment variable: +# OPENAI_API_KEY, ANTHROPIC_API_KEY, OPENROUTER_API_KEY + +providers: + openai: + enabled: false + api_key: "" # or set OPENAI_API_KEY env var + default_model: "gpt-4o" + + anthropic: + enabled: false + api_key: "" # or set ANTHROPIC_API_KEY env var + default_model: "claude-sonnet-4-6" + + openrouter: + enabled: false + api_key: "" # or set OPENROUTER_API_KEY env var + base_url: "https://openrouter.ai/api/v1" + default_model: "" # set per-session in the UI + +# Which provider to use by default for new sessions +chat: + default_provider: "openai" # openai | anthropic | openrouter + +# Separate model used for background tasks: Draft with AI, session auto-title. +# Can be a cheaper/faster model than the chat provider. +background_intelligence: + provider: "openai" + model: "gpt-4o-mini" diff --git a/backend/pyproject.toml b/backend/pyproject.toml new file mode 100644 index 0000000..a80a7b2 --- /dev/null +++ b/backend/pyproject.toml @@ -0,0 +1,44 @@ +[project] +name = "smriti-backend" +version = "0.1.0" +description = "Smriti - Cross-agent memory handoff system" +requires-python = ">=3.11" +dependencies = [ + "fastapi>=0.115.0", + "uvicorn[standard]>=0.30.0", + "sqlalchemy>=2.0.0", + "alembic>=1.13.0", + "pydantic>=2.0.0", + "pydantic-settings>=2.0.0", + "psycopg2-binary>=2.9.0", + "httpx>=0.27.0", + "openai>=1.0.0", + "anthropic>=0.40.0", + "pyyaml>=6.0", + "pgvector>=0.2.0", +] + +[project.optional-dependencies] +dev = [ + "pytest>=8.0.0", + "pytest-asyncio>=0.23.0", + "pytest-cov>=5.0.0", + "ruff>=0.4.0", +] + +[tool.setuptools.packages.find] +include = ["app*"] + +[tool.ruff] +target-version = "py311" +line-length = 100 + +[tool.ruff.lint] +select = ["E", "F", "I", "N", "W", "UP"] + +[tool.ruff.lint.isort] +known-first-party = ["app"] + +[tool.pytest.ini_options] +testpaths = ["tests"] +asyncio_mode = "auto" diff --git a/backend/tests/__init__.py b/backend/tests/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/backend/tests/conftest.py b/backend/tests/conftest.py new file mode 100644 index 0000000..00bb153 --- /dev/null +++ b/backend/tests/conftest.py @@ -0,0 +1,111 @@ +"""Shared test fixtures and configuration.""" + +import pathlib + +import pytest + +from app.domain.enums import MessageRole +from app.domain.models import ( + CodeSnippet, + Decision, + Entity, + ExtractionResult, + Message, + OpenQuestion, + Task, +) +from app.services.llm.mock_provider import MockProvider + +FIXTURES_DIR = pathlib.Path(__file__).parent / "fixtures" + + +@pytest.fixture +def fixtures_dir(): + """Path to test fixtures directory.""" + return FIXTURES_DIR + + +@pytest.fixture +def coding_transcript(fixtures_dir): + """Load coding session fixture.""" + return (fixtures_dir / "coding_session.txt").read_text() + + +@pytest.fixture +def planning_transcript(fixtures_dir): + """Load planning session fixture.""" + return (fixtures_dir / "planning_session.txt").read_text() + + +@pytest.fixture +def debugging_transcript(fixtures_dir): + """Load debugging session fixture.""" + return (fixtures_dir / "debugging_session.txt").read_text() + + +@pytest.fixture +def research_transcript(fixtures_dir): + """Load research session fixture.""" + return (fixtures_dir / "research_session.txt").read_text() + + +@pytest.fixture +def unlabeled_transcript(fixtures_dir): + """Load unlabeled session fixture.""" + return (fixtures_dir / "unlabeled_session.txt").read_text() + + +@pytest.fixture +def code_heavy_transcript(fixtures_dir): + """Load code-heavy session fixture.""" + return (fixtures_dir / "code_heavy_session.txt").read_text() + + +@pytest.fixture +def mock_provider(): + """Mock LLM provider for testing.""" + return MockProvider() + + +@pytest.fixture +def sample_messages(): + """A set of sample parsed messages for testing.""" + return [ + Message(role=MessageRole.HUMAN, content="Can you help me set up a database?", position=0), + Message(role=MessageRole.ASSISTANT, content="Sure! I'd recommend using SQLAlchemy. We decided to use PostgreSQL for this project.", position=1), + Message(role=MessageRole.HUMAN, content="What about the migration tool? We need to handle schema changes.", position=2), + Message(role=MessageRole.ASSISTANT, content="Use Alembic for migrations. TODO: set up the initial migration script.", position=3), + ] + + +@pytest.fixture +def sample_extraction_result(): + """A sample extraction result for testing pack generation.""" + return ExtractionResult( + summary="Session about setting up a database with SQLAlchemy and PostgreSQL. Discussed migration strategies using Alembic.", + decisions=[ + Decision(description="Use PostgreSQL as the database", context="Better suited for our use case"), + Decision(description="Use Alembic for migrations", context="Standard tool for SQLAlchemy"), + ], + tasks=[ + Task(description="Set up initial migration script", status="pending"), + Task(description="Configure database connection pooling", status="pending"), + Task(description="Create User model", status="completed"), + ], + open_questions=[ + OpenQuestion(question="Should we use async SQLAlchemy?", context="Performance considerations"), + ], + entities=[ + Entity(name="SQLAlchemy", type="technology", context="ORM framework"), + Entity(name="PostgreSQL", type="technology", context="Database"), + Entity(name="Alembic", type="technology", context="Migration tool"), + Entity(name="models.py", type="file", context="Database models file"), + ], + code_snippets=[ + CodeSnippet( + language="python", + code="class User(Base):\n __tablename__ = 'users'\n id = Column(Integer, primary_key=True)", + description="User model definition", + ), + ], + ) diff --git a/backend/tests/fixtures/code_heavy_session.txt b/backend/tests/fixtures/code_heavy_session.txt new file mode 100644 index 0000000..4f58cae --- /dev/null +++ b/backend/tests/fixtures/code_heavy_session.txt @@ -0,0 +1,36 @@ +Human: Can you write me a Python decorator that retries a function on failure? +Assistant: Here's a retry decorator with configurable attempts and exponential backoff: + +```python +import time +import functools + +def retry(max_attempts=3, delay=1, backoff=2): + def decorator(func): + @functools.wraps(func) + def wrapper(*args, **kwargs): + attempts = 0 + current_delay = delay + while attempts < max_attempts: + try: + return func(*args, **kwargs) + except Exception as e: + attempts += 1 + if attempts == max_attempts: + raise + time.sleep(current_delay) + current_delay *= backoff + return wrapper + return decorator +``` + +Usage: +```python +@retry(max_attempts=3, delay=0.5) +def fetch_data(url): + response = requests.get(url) + response.raise_for_status() + return response.json() +``` + +We decided to use exponential backoff by default. TODO: add logging to the retry attempts. TODO: add support for specifying which exception types to retry on. Should we also add a jitter factor to prevent thundering herd issues? diff --git a/backend/tests/fixtures/coding_session.txt b/backend/tests/fixtures/coding_session.txt new file mode 100644 index 0000000..97b06fe --- /dev/null +++ b/backend/tests/fixtures/coding_session.txt @@ -0,0 +1,52 @@ +Human: I'm building a REST API for a task management app using FastAPI and SQLAlchemy. Can you help me set up the database models? I need a User model with email and name, and a Task model with title, description, status, and a foreign key to the user. +Assistant: Sure! Here's how I'd structure those models using SQLAlchemy 2.0 with mapped_column: + +```python +from sqlalchemy import String, Text, ForeignKey +from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column, relationship + +class Base(DeclarativeBase): + pass + +class User(Base): + __tablename__ = "users" + id: Mapped[int] = mapped_column(primary_key=True) + email: Mapped[str] = mapped_column(String(255), unique=True) + name: Mapped[str] = mapped_column(String(100)) + tasks: Mapped[list["Task"]] = relationship(back_populates="owner") + +class Task(Base): + __tablename__ = "tasks" + id: Mapped[int] = mapped_column(primary_key=True) + title: Mapped[str] = mapped_column(String(200)) + description: Mapped[str | None] = mapped_column(Text, nullable=True) + status: Mapped[str] = mapped_column(String(20), default="pending") + user_id: Mapped[int] = mapped_column(ForeignKey("users.id")) + owner: Mapped["User"] = relationship(back_populates="tasks") +``` + +We decided to use mapped_column style instead of the older Column() approach since we're on SQLAlchemy 2.0. TODO: add indexes on the email and status columns for query performance. +Human: Great! What about migrations? Should we use Alembic? +Assistant: Yes, Alembic is the standard migration tool for SQLAlchemy. We decided to use Alembic with autogenerate. Here's how to set it up: + +1. Install: `pip install alembic` +2. Initialize: `alembic init alembic` +3. Configure `alembic/env.py` to import your Base metadata + +TODO: create the initial migration after finalizing the models. Should we add soft delete support to the Task model? +Human: Good question. Let's not add soft delete for the MVP. What about input validation? +Assistant: For the MVP, I'd recommend using Pydantic for request/response validation. We decided to keep it simple — no soft delete for now, just hard deletes. Here's a basic schema: + +```python +from pydantic import BaseModel, EmailStr + +class UserCreate(BaseModel): + email: EmailStr + name: str + +class TaskCreate(BaseModel): + title: str + description: str | None = None +``` + +TODO: add proper error handling middleware. One open question is whether we should add pagination to the task listing endpoint? diff --git a/backend/tests/fixtures/debugging_session.txt b/backend/tests/fixtures/debugging_session.txt new file mode 100644 index 0000000..70ee003 --- /dev/null +++ b/backend/tests/fixtures/debugging_session.txt @@ -0,0 +1 @@ +Human: I'm getting a weird error in production. The API is returning 500 errors intermittently. The logs show a connection pool exhaustion issue with the database. We decided to increase the pool size from 5 to 20. We also need to add connection timeout handling. Can you help me debug this? diff --git a/backend/tests/fixtures/planning_session.txt b/backend/tests/fixtures/planning_session.txt new file mode 100644 index 0000000..c5fe47a --- /dev/null +++ b/backend/tests/fixtures/planning_session.txt @@ -0,0 +1,12 @@ +Human: Let's plan the next quarter product roadmap. We need to decide on priorities for Q2. The team is 5 engineers and we have too many things to do. +Assistant: Let me help structure this. First, let's identify the key areas. What are the main candidates for Q2? +Human: We have: 1) Mobile app launch, 2) API v2 migration, 3) Performance improvements, 4) Dashboard redesign, 5) Analytics pipeline. We decided to cut analytics pipeline from Q2 — it's important but not urgent. +Assistant: Good call. For 5 engineers over 3 months, I'd suggest focusing on 2-3 major initiatives. We decided to prioritize Mobile app launch and API v2 migration as the top two. The Dashboard redesign can be a stretch goal. + +Key decisions so far: +- Mobile app launch is P0 for Q2 +- API v2 migration is P1 +- Analytics pipeline deferred to Q3 +- Dashboard redesign is a stretch goal + +TODO: create detailed project plans for each initiative. TODO: assign engineering leads to each project. Should we bring in a contractor for the mobile app, or keep it all in-house? That's an open question we need to resolve by end of week. diff --git a/backend/tests/fixtures/research_session.txt b/backend/tests/fixtures/research_session.txt new file mode 100644 index 0000000..0254b12 --- /dev/null +++ b/backend/tests/fixtures/research_session.txt @@ -0,0 +1 @@ +Human: I'm researching different approaches to implement real-time notifications in a web app. What are the main options? I'm considering WebSockets vs Server-Sent Events vs long polling. diff --git a/backend/tests/fixtures/unlabeled_session.txt b/backend/tests/fixtures/unlabeled_session.txt new file mode 100644 index 0000000..ac58fe9 --- /dev/null +++ b/backend/tests/fixtures/unlabeled_session.txt @@ -0,0 +1 @@ +So I was thinking about the architecture for this new service. It needs to handle about 10,000 requests per second and we probably want to use a message queue for the async processing. RabbitMQ seems like overkill, maybe Redis Streams would work. The main concern is that we need exactly-once processing semantics, and Redis Streams support consumer groups which could help. We decided to go with Redis Streams for the MVP and potentially migrate to Kafka later if we need better durability guarantees. The TODO list is: set up Redis Streams consumer, implement the worker process, add dead letter queue handling, and write integration tests. One open question is whether we should use a separate Redis instance for the queue or share the caching Redis instance. diff --git a/backend/tests/golden/pack_chatgpt.json b/backend/tests/golden/pack_chatgpt.json new file mode 100644 index 0000000..a3a23f8 --- /dev/null +++ b/backend/tests/golden/pack_chatgpt.json @@ -0,0 +1,6 @@ +{ + "target_tool": "chatgpt", + "format": "markdown", + "content_length": 1075, + "content": "I'm continuing a previous work session. Here's the context you need to help me pick up where I left off:\n\n**What we were working on:** Session about database setup with SQLAlchemy and Alembic. Covered User and Task models, migration strategy, and input validation with Pydantic.\n\n**Key decisions already made:**\n- Use SQLAlchemy 2.0 mapped_column style\n- Use Alembic with autogenerate for migrations\n- No soft delete for MVP\n\n**Outstanding tasks:**\n- Add indexes on email and status columns (status: pending)\n- Create initial migration (status: pending)\n- Add error handling middleware (status: pending)\n\n**Open questions to address:**\n- Should we add pagination to the task listing endpoint?\n\n**Important context:**\nKey entities/concepts: SQLAlchemy, Alembic, Pydantic, models.py\n\n**Relevant code from the session:**\n```python\nclass User(Base):\n __tablename__ = 'users'\n id: Mapped[int] = mapped_column(primary_key=True)\n```\n(User model)\n\nPlease help me continue this work. Start by confirming you understand the context, then let's proceed with the outstanding tasks." +} diff --git a/backend/tests/golden/pack_claude.json b/backend/tests/golden/pack_claude.json new file mode 100644 index 0000000..e76be25 --- /dev/null +++ b/backend/tests/golden/pack_claude.json @@ -0,0 +1,6 @@ +{ + "target_tool": "claude", + "format": "markdown", + "content_length": 1270, + "content": "I'm continuing work from a previous session. Here's the structured context:\n\n\nSession about database setup with SQLAlchemy and Alembic. Covered User and Task models, migration strategy, and input validation with Pydantic.\n\n\n Use SQLAlchemy 2.0 mapped_column style\n Use Alembic with autogenerate for migrations\n No soft delete for MVP\n\n\n\n Add indexes on email and status columns\n Create initial migration\n Add error handling middleware\n\n\n\n Should we add pagination to the task listing endpoint?\n\n\n\n SQLAlchemy\n Alembic\n Pydantic\n models.py\n\n\n\n \n class User(Base):\n __tablename__ = 'users'\n id: Mapped[int] = mapped_column(primary_key=True)\n \n\n\n\n\nPlease review this context and help me continue. Focus on the outstanding tasks and open questions." +} diff --git a/backend/tests/golden/pack_cursor.json b/backend/tests/golden/pack_cursor.json new file mode 100644 index 0000000..bb21f0f --- /dev/null +++ b/backend/tests/golden/pack_cursor.json @@ -0,0 +1,6 @@ +{ + "target_tool": "cursor", + "format": "markdown", + "content_length": 744, + "content": "# Continuation Context\n\n## Summary\nSession about database setup with SQLAlchemy and Alembic. Covered User and Task models, migration strategy, and input validation with Pydantic.\n\n## Task Checklist\n- [ ] Add indexes on email and status columns\n- [ ] Create initial migration\n- [ ] Add error handling middleware\n\n## Decisions\n- Use SQLAlchemy 2.0 mapped_column style\n- Use Alembic with autogenerate for migrations\n- No soft delete for MVP\n\n## Relevant Files\n- `models.py`\n\n## Technologies\n- SQLAlchemy\n- Alembic\n- Pydantic\n\n## Code Context\n### User model\n```python\nclass User(Base):\n __tablename__ = 'users'\n id: Mapped[int] = mapped_column(primary_key=True)\n```\n\n## Open Questions\n- Should we add pagination to the task listing endpoint?\n" +} diff --git a/backend/tests/golden/pack_generic.json b/backend/tests/golden/pack_generic.json new file mode 100644 index 0000000..c3c6a74 --- /dev/null +++ b/backend/tests/golden/pack_generic.json @@ -0,0 +1,6 @@ +{ + "target_tool": "generic", + "format": "markdown", + "content_length": 939, + "content": "# Session Continuation Pack\n\n## Summary\nSession about database setup with SQLAlchemy and Alembic. Covered User and Task models, migration strategy, and input validation with Pydantic.\n\n## Key Decisions\n- **Use SQLAlchemy 2.0 mapped_column style**\n Context: Modern approach\n- **Use Alembic with autogenerate for migrations**\n Context: Standard SQLAlchemy migration tool\n- **No soft delete for MVP**\n Context: Keep it simple\n\n## Tasks\n- ⬜ Add indexes on email and status columns [pending]\n- ⬜ Create initial migration [pending]\n- ⬜ Add error handling middleware [pending]\n\n## Open Questions\n- Should we add pagination to the task listing endpoint?\n Context: API design\n\n## Entities\n- **SQLAlchemy** (technology)\n- **Alembic** (technology)\n- **Pydantic** (technology)\n- **models.py** (file)\n\n## Code Snippets\n### User model\n```python\nclass User(Base):\n __tablename__ = 'users'\n id: Mapped[int] = mapped_column(primary_key=True)\n```\n" +} diff --git a/backend/tests/golden/parser_code_heavy_session.json b/backend/tests/golden/parser_code_heavy_session.json new file mode 100644 index 0000000..16cd6cc --- /dev/null +++ b/backend/tests/golden/parser_code_heavy_session.json @@ -0,0 +1,15 @@ +{ + "message_count": 2, + "roles": [ + "human", + "assistant" + ], + "positions": [ + 0, + 1 + ], + "content_lengths": [ + 71, + 1120 + ] +} diff --git a/backend/tests/golden/parser_coding_session.json b/backend/tests/golden/parser_coding_session.json new file mode 100644 index 0000000..308e6fd --- /dev/null +++ b/backend/tests/golden/parser_coding_session.json @@ -0,0 +1,27 @@ +{ + "message_count": 6, + "roles": [ + "human", + "assistant", + "human", + "assistant", + "human", + "assistant" + ], + "positions": [ + 0, + 1, + 2, + 3, + 4, + 5 + ], + "content_lengths": [ + 245, + 1163, + 52, + 379, + 82, + 500 + ] +} diff --git a/backend/tests/golden/parser_planning_session.json b/backend/tests/golden/parser_planning_session.json new file mode 100644 index 0000000..65a6385 --- /dev/null +++ b/backend/tests/golden/parser_planning_session.json @@ -0,0 +1,21 @@ +{ + "message_count": 4, + "roles": [ + "human", + "assistant", + "human", + "assistant" + ], + "positions": [ + 0, + 1, + 2, + 3 + ], + "content_lengths": [ + 143, + 101, + 204, + 612 + ] +} diff --git a/backend/tests/golden/parser_unlabeled_session.json b/backend/tests/golden/parser_unlabeled_session.json new file mode 100644 index 0000000..864ab83 --- /dev/null +++ b/backend/tests/golden/parser_unlabeled_session.json @@ -0,0 +1,12 @@ +{ + "message_count": 1, + "roles": [ + "unknown" + ], + "positions": [ + 0 + ], + "content_lengths": [ + 762 + ] +} diff --git a/backend/tests/integration/__init__.py b/backend/tests/integration/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/backend/tests/integration/conftest.py b/backend/tests/integration/conftest.py new file mode 100644 index 0000000..ce8a659 --- /dev/null +++ b/backend/tests/integration/conftest.py @@ -0,0 +1,104 @@ +"""Integration test configuration — uses SQLite in-memory database. + +This conftest overrides the FastAPI app's `get_db` dependency to use +a fresh in-memory SQLite database for each test session, enabling +full API integration tests without requiring PostgreSQL. + +JSONB columns are mapped to JSON (TEXT-backed) for SQLite compatibility. +""" + +import pytest +from fastapi.testclient import TestClient +from sqlalchemy import JSON, create_engine, event +from sqlalchemy.dialects.postgresql import JSONB +from sqlalchemy.orm import sessionmaker +from sqlalchemy.pool import StaticPool + +from app.db.database import Base, get_db +from app.main import app + + +@pytest.fixture(scope="function") +def db_engine(): + engine = create_engine( + "sqlite://", + connect_args={"check_same_thread": False}, + poolclass=StaticPool, + ) + + # Enable foreign keys in SQLite + @event.listens_for(engine, "connect") + def set_sqlite_pragma(dbapi_connection, connection_record): + cursor = dbapi_connection.cursor() + cursor.execute("PRAGMA foreign_keys=ON") + cursor.close() + + # Map JSONB → JSON for SQLite (JSONB is Postgres-only) + # We temporarily swap the impl so create_all works on SQLite + _render_original = JSONB().compile + + @event.listens_for(engine, "before_cursor_execute", retval=True) + def receive_before_cursor_execute(conn, cursor, statement, parameters, context, executemany): + # Replace JSONB with JSON in DDL statements for SQLite + if "JSONB" in statement: + statement = statement.replace("JSONB", "JSON") + if "VECTOR" in statement: + statement = statement.replace("VECTOR", "JSON") + return statement, parameters + + # Create tables, mapping JSONB and Vector for SQLite + from sqlalchemy.dialects import sqlite as sqlite_dialect + from sqlalchemy.dialects.sqlite.base import SQLiteTypeCompiler + from pgvector.sqlalchemy import Vector + + original_visit_jsonb = getattr(SQLiteTypeCompiler, "visit_JSONB", None) + original_visit_vector = getattr(SQLiteTypeCompiler, "visit_vector", None) + + SQLiteTypeCompiler.visit_JSONB = lambda self, type_, **kw: "JSON" + SQLiteTypeCompiler.visit_vector = lambda self, type_, **kw: "JSON" + + Base.metadata.create_all(bind=engine) + + # Restore + if original_visit_jsonb: + SQLiteTypeCompiler.visit_JSONB = original_visit_jsonb + else: + try: + del SQLiteTypeCompiler.visit_JSONB + except AttributeError: + pass + + if original_visit_vector: + SQLiteTypeCompiler.visit_vector = original_visit_vector + else: + try: + del SQLiteTypeCompiler.visit_vector + except AttributeError: + pass + + yield engine + Base.metadata.drop_all(bind=engine) + + +@pytest.fixture(scope="function") +def db_session(db_engine): + SessionLocal = sessionmaker(autocommit=False, autoflush=False, bind=db_engine) + session = SessionLocal() + yield session + session.close() + + +@pytest.fixture(scope="function") +def client(db_session): + """FastAPI TestClient with overridden DB dependency.""" + + def override_get_db(): + try: + yield db_session + finally: + pass + + app.dependency_overrides[get_db] = override_get_db + with TestClient(app) as c: + yield c + app.dependency_overrides.clear() diff --git a/backend/tests/integration/test_api_v2_git.py b/backend/tests/integration/test_api_v2_git.py new file mode 100644 index 0000000..6635a77 --- /dev/null +++ b/backend/tests/integration/test_api_v2_git.py @@ -0,0 +1,157 @@ +import pytest + +def test_repo_lifecycle(client): + # 1. Create Repo + repo_resp = client.post("/api/v2/repos", json={ + "name": "Integration Test Repo", + "description": "Repo testing" + }) + assert repo_resp.status_code == 201 + repo_id = repo_resp.json()["id"] + assert repo_resp.json()["name"] == "Integration Test Repo" + + # 2. Get Repo + get_resp = client.get(f"/api/v2/repos/{repo_id}") + assert get_resp.status_code == 200 + assert get_resp.json()["name"] == "Integration Test Repo" + + # 3. List Repos + list_resp = client.get("/api/v2/repos") + assert list_resp.status_code == 200 + assert len(list_resp.json()) >= 1 + assert any(r["id"] == repo_id for r in list_resp.json()) + + +def test_commit_lifecycle(client): + # Setup repo + repo_resp = client.post("/api/v2/repos", json={"name": "Commit Test Repo"}) + repo_id = repo_resp.json()["id"] + + # 1. Create first commit + c1_resp = client.post("/api/v2/commits", json={ + "repo_id": repo_id, + "author_agent": "chatgpt", + "author_type": "llm", + "message": "Init", + "objective": "Start project" + }) + assert c1_resp.status_code == 201 + c1_id = c1_resp.json()["id"] + assert c1_resp.json()["parent_commit_id"] is None + + # 2. Create child commit + c2_resp = client.post("/api/v2/commits", json={ + "repo_id": repo_id, + "parent_commit_id": c1_id, + "author_agent": "claude", + "author_type": "llm", + "message": "Update 1", + "objective": "Continue project" + }) + assert c2_resp.status_code == 201 + c2_id = c2_resp.json()["id"] + assert c2_resp.json()["parent_commit_id"] == c1_id + + # 3. Get commit explicitly + get_c1 = client.get(f"/api/v2/commits/{c1_id}") + assert get_c1.status_code == 200 + assert get_c1.json()["message"] == "Init" + + # 4. List repo commits + hist_resp = client.get(f"/api/v2/repos/{repo_id}/commits") + assert hist_resp.status_code == 200 + hist = hist_resp.json() + assert len(hist) == 2 + # Ensure descending order + assert hist[0]["id"] == c2_id + assert hist[1]["id"] == c1_id + + # 5. Get latest commit + latest_resp = client.get(f"/api/v2/repos/{repo_id}/commits/latest") + assert latest_resp.status_code == 200 + assert latest_resp.json()["id"] == c2_id + + +def test_context_from_commit(client): + # Setup repo and commit + repo_resp = client.post("/api/v2/repos", json={"name": "Context Test Repo"}) + repo_id = repo_resp.json()["id"] + + c1_resp = client.post("/api/v2/commits", json={ + "repo_id": repo_id, + "author_agent": "user", + "author_type": "user", + "message": "Setup memory", + "summary": "This is a summary text.", + "tasks": ["Task A", "Task B"], + "decisions": ["Decision 1"] + }) + c1_id = c1_resp.json()["id"] + + # Generate Generic payload + ctx_resp = client.post("/api/v2/context/from-commit", json={ + "commit_id": c1_id, + "target": "generic" + }) + assert ctx_resp.status_code == 200 + content = ctx_resp.json()["content"] + assert "Context Test Repo" in content + assert "This is a summary text." in content + assert "Task A" in content + assert "Decision 1" in content + + # Generate Claude payload + ctx_resp_claude = client.post("/api/v2/context/from-commit", json={ + "commit_id": c1_id, + "target": "claude" + }) + assert ctx_resp_claude.status_code == 200 + content_claude = ctx_resp_claude.json()["content"] + assert "" in content_claude + assert "This is a summary text." in content_claude + + +def test_parent_delta(client): + """parent-delta for root commit returns null parent; child commit returns populated parent.""" + # Setup + repo_resp = client.post("/api/v2/repos", json={"name": "Delta Test Repo"}) + repo_id = repo_resp.json()["id"] + + # Root commit + c1_resp = client.post("/api/v2/commits", json={ + "repo_id": repo_id, + "message": "Root state", + "tasks": ["Task A"], + "decisions": ["Decision A"], + }) + assert c1_resp.status_code == 201 + c1_id = c1_resp.json()["id"] + + # Root delta — parent should be null + delta_root = client.get(f"/api/v2/context/parent-delta/{c1_id}") + assert delta_root.status_code == 200 + root_data = delta_root.json() + assert root_data["current"]["id"] == c1_id + assert root_data["parent"] is None + + # Child commit with different tasks + c2_resp = client.post("/api/v2/commits", json={ + "repo_id": repo_id, + "parent_commit_id": c1_id, + "message": "Second state", + "tasks": ["Task A", "Task B"], + "decisions": ["Decision A", "Decision B"], + }) + assert c2_resp.status_code == 201 + c2_id = c2_resp.json()["id"] + + # Child delta — parent should have c1 data + delta_child = client.get(f"/api/v2/context/parent-delta/{c2_id}") + assert delta_child.status_code == 200 + child_data = delta_child.json() + assert child_data["current"]["id"] == c2_id + assert child_data["parent"]["id"] == c1_id + assert "Task A" in child_data["parent"]["tasks"] + assert "Task B" not in child_data["parent"]["tasks"] + assert "Task B" in child_data["current"]["tasks"] + diff --git a/backend/tests/integration/test_api_v4_chat.py b/backend/tests/integration/test_api_v4_chat.py new file mode 100644 index 0000000..921e023 --- /dev/null +++ b/backend/tests/integration/test_api_v4_chat.py @@ -0,0 +1,206 @@ +""" +V4 Chat API integration tests. +All tests use use_mock=True so they run without any real provider API keys. +""" +import pytest + + +def test_provider_status(client): + """Providers endpoint returns status dict.""" + resp = client.get("/api/v4/chat/providers") + assert resp.status_code == 200 + data = resp.json() + assert {"openai", "anthropic", "openrouter"}.issubset(set(data.keys())) + for _name, status in data.items(): + assert "enabled" in status + assert "has_key" in status + + +def _create_repo(client, name="Chat Test Repo"): + r = client.post("/api/v2/repos", json={"name": name}) + assert r.status_code == 201, r.text + return r.json()["id"] + + +def test_session_lifecycle(client): + repo_id = _create_repo(client) + + # Create session (no head commit yet) + s = client.post(f"/api/v4/chat/spaces/{repo_id}/sessions", json={ + "title": "Test session", + "provider": "openrouter", + "model": "mock-model", + "seed_from": "head", + }) + assert s.status_code == 201, s.text + session = s.json() + assert session["repo_id"] == repo_id + assert session["active_provider"] == "openrouter" + assert session["seeded_commit_id"] is None # No head commit exists yet + + session_id = session["id"] + + # Fetch session + get_s = client.get(f"/api/v4/chat/spaces/{repo_id}/sessions/{session_id}") + assert get_s.status_code == 200 + assert get_s.json()["id"] == session_id + + # Initially no turns + turns = client.get(f"/api/v4/chat/spaces/{repo_id}/sessions/{session_id}/turns") + assert turns.status_code == 200 + assert turns.json() == [] + + +def test_send_message_mock(client): + repo_id = _create_repo(client, "Send Test Repo") + + s = client.post(f"/api/v4/chat/spaces/{repo_id}/sessions", json={ + "title": "Send test", + "provider": "openrouter", + "model": "mock-model", + }) + session_id = s.json()["id"] + + # Send a message using the mock adapter + send = client.post("/api/v4/chat/send", json={ + "repo_id": repo_id, + "session_id": session_id, + "provider": "openrouter", + "model": "mock-model", + "message": "What is the plan?", + "use_mock": True, + }) + assert send.status_code == 200, send.text + resp = send.json() + assert "reply" in resp + assert "mock:mock-model" in resp["reply"] + assert "What is the plan?" in resp["reply"] + assert resp["turn_count"] == 2 # user + assistant + + # Turns should now exist + turns = client.get(f"/api/v4/chat/spaces/{repo_id}/sessions/{session_id}/turns") + assert turns.status_code == 200 + turn_list = turns.json() + assert len(turn_list) == 2 + assert turn_list[0]["role"] == "user" + assert turn_list[1]["role"] == "assistant" + + +def test_model_switch_preserves_history(client): + """Switching provider/model mid-session keeps prior turns in context.""" + repo_id = _create_repo(client, "Model Switch Repo") + + s = client.post(f"/api/v4/chat/spaces/{repo_id}/sessions", json={ + "provider": "openrouter", "model": "mock-model", + }) + session_id = s.json()["id"] + + # First turn with mock/openrouter + client.post("/api/v4/chat/send", json={ + "repo_id": repo_id, "session_id": session_id, + "provider": "openrouter", "model": "gpt-mock", + "message": "Turn one", "use_mock": True, + }) + + # Second turn — simulate model switch (different model string) + send2 = client.post("/api/v4/chat/send", json={ + "repo_id": repo_id, "session_id": session_id, + "provider": "anthropic", "model": "claude-mock", + "message": "Turn two, new model", "use_mock": True, + }) + assert send2.status_code == 200 + + # All 4 turns should exist (2 user + 2 assistant) + turns = client.get(f"/api/v4/chat/spaces/{repo_id}/sessions/{session_id}/turns") + assert len(turns.json()) == 4 + + # Session should reflect latest model + sess = client.get(f"/api/v4/chat/spaces/{repo_id}/sessions/{session_id}") + assert sess.json()["active_provider"] == "anthropic" + assert sess.json()["active_model"] == "claude-mock" + + +def test_manual_commit_from_session(client): + repo_id = _create_repo(client, "Commit Session Repo") + + s = client.post(f"/api/v4/chat/spaces/{repo_id}/sessions", json={ + "provider": "openrouter", "model": "mock", + }) + session_id = s.json()["id"] + + # Chat first + client.post("/api/v4/chat/send", json={ + "repo_id": repo_id, "session_id": session_id, + "provider": "openrouter", "model": "mock", + "message": "We decided to use Postgres", "use_mock": True, + }) + + # Manual commit + commit_r = client.post("/api/v4/chat/commit", json={ + "repo_id": repo_id, + "session_id": session_id, + "message": "Decided on database", + "summary": "We chose Postgres as our primary DB.", + "decisions": ["Use Postgres", "No ORM for now"], + "tasks": ["Set up schema"], + }) + assert commit_r.status_code == 201, commit_r.text + commit = commit_r.json() + assert commit["message"] == "Decided on database" + assert len(commit["commit_hash"]) == 64 + + +def test_head_endpoint(client): + repo_id = _create_repo(client, "Head Repo") + + # No session yet + head = client.get(f"/api/v4/chat/spaces/{repo_id}/head") + assert head.status_code == 200 + data = head.json() + assert data["commit_hash"] is None + assert data["latest_session_id"] is None + + # Create session + commit + s = client.post(f"/api/v4/chat/spaces/{repo_id}/sessions", json={"provider": "openrouter", "model": "mock"}) + session_id = s.json()["id"] + + client.post("/api/v4/chat/commit", json={ + "repo_id": repo_id, "session_id": session_id, + "message": "First commit", "objective": "Test head", + }) + + head2 = client.get(f"/api/v4/chat/spaces/{repo_id}/head") + assert head2.status_code == 200 + data2 = head2.json() + assert data2["commit_hash"] is not None + assert data2["objective"] == "Test head" + assert data2["latest_session_id"] == session_id + + +def test_session_seeded_from_head(client): + """Session created after a commit should have seeded_commit_id set.""" + repo_id = _create_repo(client, "Seed Test Repo") + + # Create a session and commit first + s = client.post(f"/api/v4/chat/spaces/{repo_id}/sessions", json={"provider": "openrouter", "model": "mock"}) + session_id = s.json()["id"] + + commit_r = client.post("/api/v4/chat/commit", json={ + "repo_id": repo_id, "session_id": session_id, + "message": "Initial state", "summary": "We started here.", + }) + commit_id = commit_r.json()["id"] + + # Now open a NEW session — should be seeded from the commit + s2 = client.post(f"/api/v4/chat/spaces/{repo_id}/sessions", json={ + "provider": "openrouter", "model": "mock", "seed_from": "head", + }) + assert s2.json()["seeded_commit_id"] == commit_id + + # First send should inject seed context (just verify it doesn't crash) + send = client.post("/api/v4/chat/send", json={ + "repo_id": repo_id, "session_id": s2.json()["id"], + "provider": "openrouter", "model": "mock", + "message": "Continue from prior state", "use_mock": True, + }) + assert send.status_code == 200 diff --git a/backend/tests/integration/test_api_v5_lineage.py b/backend/tests/integration/test_api_v5_lineage.py new file mode 100644 index 0000000..3d36868 --- /dev/null +++ b/backend/tests/integration/test_api_v5_lineage.py @@ -0,0 +1,616 @@ +""" +V5 Lineage API integration tests. + +All chat turns use use_mock=True to avoid requiring real provider API keys. + +Covers: +- Fork session creation and field correctness +- Forked session context isolation (the critical regression test) +- Lineage endpoint shape +- Compare endpoint diffs +- Auto-inherit of forked_from_checkpoint_id in send_message +- Error cases +""" +import pytest + + +# ── Helpers ─────────────────────────────────────────────────────────────────── + +def _create_repo(client, name="Lineage Test Repo"): + r = client.post("/api/v2/repos", json={"name": name}) + assert r.status_code == 201, r.text + return r.json()["id"] + + +def _create_session(client, repo_id, title="test session"): + r = client.post(f"/api/v4/chat/spaces/{repo_id}/sessions", json={ + "title": title, "provider": "openrouter", "model": "mock", + }) + assert r.status_code == 201, r.text + return r.json()["id"] + + +def _send(client, session_id, message="hello"): + r = client.post("/api/v4/chat/send", json={ + "session_id": session_id, + "provider": "openrouter", + "model": "mock", + "message": message, + "use_mock": True, + }) + assert r.status_code == 200, r.text + return r.json() + + +def _commit(client, repo_id, session_id, message="checkpoint", **kwargs): + payload = { + "repo_id": repo_id, + "session_id": session_id, + "message": message, + **kwargs, + } + r = client.post("/api/v4/chat/commit", json=payload) + assert r.status_code == 201, r.text + return r.json() + + +def _fork(client, space_id, checkpoint_id, branch_name="", provider="", model=""): + r = client.post("/api/v5/lineage/sessions/fork", json={ + "space_id": space_id, + "checkpoint_id": checkpoint_id, + "branch_name": branch_name, + "provider": provider, + "model": model, + }) + return r + + +# ── Fork tests ──────────────────────────────────────────────────────────────── + +def test_fork_creates_new_session(client): + """POST fork returns a new session with correct fork fields.""" + repo_id = _create_repo(client, "Fork Basic") + session_id = _create_session(client, repo_id) + commit = _commit(client, repo_id, session_id, "base commit", + summary="Base state.", decisions=["Use SQLite"]) + checkpoint_id = commit["id"] + + r = _fork(client, repo_id, checkpoint_id, branch_name="experiment-1") + assert r.status_code == 201, r.text + data = r.json() + + assert data["forked_from_checkpoint_id"] == checkpoint_id + assert data["branch_name"] == "experiment-1" + assert data["history_base_seq"] == 0 + # Must be a fresh UUID, not the original session + assert data["session_id"] != session_id + + +def test_fork_auto_generates_branch_name(client): + """Fork with no branch_name still produces a non-empty branch name.""" + repo_id = _create_repo(client, "Fork AutoName") + session_id = _create_session(client, repo_id) + commit = _commit(client, repo_id, session_id, "base") + checkpoint_id = commit["id"] + + r = _fork(client, repo_id, checkpoint_id) + assert r.status_code == 201, r.text + assert r.json()["branch_name"] != "" + + +def test_fork_session_appears_in_session_get(client): + """GET /sessions/{id} on a forked session returns branch identity fields.""" + repo_id = _create_repo(client, "Fork Get") + session_id = _create_session(client, repo_id) + commit = _commit(client, repo_id, session_id, "base") + checkpoint_id = commit["id"] + + fork_resp = _fork(client, repo_id, checkpoint_id, branch_name="feat-branch") + fork_id = fork_resp.json()["session_id"] + + get_r = client.get(f"/api/v4/chat/sessions/{fork_id}") + assert get_r.status_code == 200, get_r.text + s = get_r.json() + assert s["forked_from_checkpoint_id"] == checkpoint_id + assert s["branch_name"] == "feat-branch" + + +def test_fork_nonexistent_checkpoint(client): + """Forking from a non-existent checkpoint returns 404.""" + repo_id = _create_repo(client, "Fork 404") + fake_id = "00000000-0000-0000-0000-000000000099" + r = _fork(client, repo_id, fake_id) + assert r.status_code == 404, r.text + + +def test_fork_checkpoint_wrong_space(client): + """Forking a checkpoint that belongs to a different space returns 400.""" + repo_a = _create_repo(client, "Space A") + repo_b = _create_repo(client, "Space B") + + session_a = _create_session(client, repo_a) + commit_a = _commit(client, repo_a, session_a, "commit in A") + checkpoint_a_id = commit_a["id"] + + # Try to fork checkpoint from space A into space B + r = _fork(client, repo_b, checkpoint_a_id) + assert r.status_code == 400, r.text + + +# ── Context isolation tests ─────────────────────────────────────────────────── + +def test_fork_isolation_between_two_forks(client): + """Two forks from the same checkpoint accumulate turns independently.""" + repo_id = _create_repo(client, "Fork Isolation") + session_id = _create_session(client, repo_id) + commit = _commit(client, repo_id, session_id, "shared base") + checkpoint_id = commit["id"] + + fork1_id = _fork(client, repo_id, checkpoint_id, branch_name="branch-1").json()["session_id"] + fork2_id = _fork(client, repo_id, checkpoint_id, branch_name="branch-2").json()["session_id"] + + _send(client, fork1_id, "Message only in branch-1") + _send(client, fork2_id, "Message only in branch-2") + + turns1 = client.get(f"/api/v4/chat/spaces/{repo_id}/sessions/{fork1_id}/turns").json() + turns2 = client.get(f"/api/v4/chat/spaces/{repo_id}/sessions/{fork2_id}/turns").json() + + branch1_contents = [t["content"] for t in turns1] + branch2_contents = [t["content"] for t in turns2] + + assert any("branch-1" in c for c in branch1_contents) + assert not any("branch-2" in c for c in branch1_contents) + assert any("branch-2" in c for c in branch2_contents) + assert not any("branch-1" in c for c in branch2_contents) + + +def test_fork_isolation_regression_checkpoint_a_does_not_see_b_era_context(client): + """ + Regression test for the commit-isolation bug. + + Scenario: + 1. Create Checkpoint A (Australia trip planning). + 2. Continue the session, add NZ context, create Checkpoint B. + 3. Fork a new session from Checkpoint A. + 4. Send a message in the fork. + 5. Verify the forked session's turns do NOT include any NZ-era turns + from the period between A and B. + + Before the fix, history_base_seq=0 was not enforced for forked sessions, + so all turns in the original session (including post-A NZ turns) bled in. + """ + repo_id = _create_repo(client, "Regression Fork") + session_id = _create_session(client, repo_id) + + # ── Phase 1: build up to Checkpoint A ──────────────────────────────────── + _send(client, session_id, "Let's plan the Australia trip.") + _send(client, session_id, "We should visit Sydney and Melbourne.") + + commit_a = _commit(client, repo_id, session_id, "Checkpoint A", + summary="Planning Australia trip.", + decisions=["Visit Sydney", "Visit Melbourne"]) + checkpoint_a_id = commit_a["id"] + + # ── Phase 2: continue in the same session, add NZ content ───────────────── + _send(client, session_id, "Actually, let's also add New Zealand to the trip.") + _send(client, session_id, "We should visit Auckland and Queenstown.") + + _commit(client, repo_id, session_id, "Checkpoint B", + summary="Added New Zealand to itinerary.", + decisions=["Visit Auckland", "Visit Queenstown"]) + + # ── Phase 3: fork from Checkpoint A ────────────────────────────────────── + fork_resp = _fork(client, repo_id, checkpoint_a_id, branch_name="australia-only") + assert fork_resp.status_code == 201, fork_resp.text + fork_id = fork_resp.json()["session_id"] + + # ── Phase 4: send a message in the fork ────────────────────────────────── + send_resp = _send(client, fork_id, "What cities are we visiting?") + + # ── Phase 5: verify NZ-era turns are not in the fork's history ─────────── + turns = client.get(f"/api/v4/chat/spaces/{repo_id}/sessions/{fork_id}/turns").json() + all_fork_content = " ".join(t["content"] for t in turns) + + # The fork should not contain NZ references from the original session's + # post-A turns. (The mock LLM echoes the prompt, so if NZ content were + # injected as history it would appear in the assistant reply.) + assert "New Zealand" not in all_fork_content, ( + "NZ-era context from post-A turns leaked into the forked session. " + "history_base_seq isolation is broken." + ) + assert "Auckland" not in all_fork_content, ( + "Auckland turn from post-A leaked into the fork." + ) + + +def test_send_auto_inherits_fork_context_without_explicit_mount(client): + """ + send_message in a forked session must use checkpoint context automatically, + even when mounted_checkpoint_id is not passed in the request. + """ + repo_id = _create_repo(client, "Fork Auto-Inherit") + session_id = _create_session(client, repo_id) + commit = _commit(client, repo_id, session_id, "base", + summary="The project uses Rust.", decisions=["Use Rust"]) + checkpoint_id = commit["id"] + + fork_id = _fork(client, repo_id, checkpoint_id).json()["session_id"] + + # Send WITHOUT passing mounted_checkpoint_id — backend must auto-inherit + r = client.post("/api/v4/chat/send", json={ + "session_id": fork_id, + "provider": "openrouter", + "model": "mock", + "message": "What language are we using?", + "use_mock": True, + # No mounted_checkpoint_id, no history_base_seq + }) + assert r.status_code == 200, r.text + + +# ── Lineage view tests ──────────────────────────────────────────────────────── + +def test_lineage_empty_space(client): + """Lineage on a space with no checkpoints or sessions returns empty lists.""" + repo_id = _create_repo(client, "Empty Lineage") + r = client.get(f"/api/v5/lineage/spaces/{repo_id}") + assert r.status_code == 200, r.text + data = r.json() + assert data["space_id"] == repo_id + assert data["checkpoints"] == [] + assert data["sessions"] == [] + + +def test_lineage_linear_chain(client): + """A → B → C checkpoint chain appears with correct parent_checkpoint_id links.""" + repo_id = _create_repo(client, "Linear Lineage") + session_id = _create_session(client, repo_id) + + _send(client, session_id, "Step 1") + commit_a = _commit(client, repo_id, session_id, "A") + + _send(client, session_id, "Step 2") + commit_b = _commit(client, repo_id, session_id, "B") + + _send(client, session_id, "Step 3") + commit_c = _commit(client, repo_id, session_id, "C") + + r = client.get(f"/api/v5/lineage/spaces/{repo_id}") + assert r.status_code == 200, r.text + checkpoints = r.json()["checkpoints"] + assert len(checkpoints) == 3 + + by_id = {c["id"]: c for c in checkpoints} + assert by_id[commit_a["id"]]["parent_checkpoint_id"] is None + assert by_id[commit_b["id"]]["parent_checkpoint_id"] == commit_a["id"] + assert by_id[commit_c["id"]]["parent_checkpoint_id"] == commit_b["id"] + + +def test_lineage_with_fork_shows_both_sessions(client): + """After forking, the lineage view contains both the original and forked sessions.""" + repo_id = _create_repo(client, "Fork Lineage") + session_id = _create_session(client, repo_id) + commit = _commit(client, repo_id, session_id, "base") + checkpoint_id = commit["id"] + + fork_resp = _fork(client, repo_id, checkpoint_id, branch_name="side-branch") + fork_id = fork_resp.json()["session_id"] + + r = client.get(f"/api/v5/lineage/spaces/{repo_id}") + assert r.status_code == 200, r.text + data = r.json() + + session_ids = {s["id"] for s in data["sessions"]} + assert session_id in session_ids + assert fork_id in session_ids + + fork_node = next(s for s in data["sessions"] if s["id"] == fork_id) + assert fork_node["forked_from_checkpoint_id"] == checkpoint_id + assert fork_node["branch_name"] == "side-branch" + + +# ── Compare tests ───────────────────────────────────────────────────────────── + +def test_compare_same_checkpoint(client): + """Comparing a checkpoint with itself produces an empty diff.""" + repo_id = _create_repo(client, "Compare Same") + session_id = _create_session(client, repo_id) + commit = _commit(client, repo_id, session_id, "base", + summary="Hello.", + decisions=["Use Postgres"], + tasks=["Set up schema"]) + cid = commit["id"] + + r = client.get(f"/api/v5/lineage/checkpoints/{cid}/compare/{cid}") + assert r.status_code == 200, r.text + diff = r.json()["diff"] + + assert diff["summary_a"] == diff["summary_b"] + assert diff["decisions_only_a"] == [] + assert diff["decisions_only_b"] == [] + assert diff["tasks_only_a"] == [] + assert diff["tasks_only_b"] == [] + assert "Use Postgres" in diff["decisions_shared"] + + +def test_compare_diverged_checkpoints(client): + """Two checkpoints with different decisions produce correct A-only / B-only / shared splits.""" + repo_id = _create_repo(client, "Compare Diverged") + session_id = _create_session(client, repo_id) + + commit_a = _commit(client, repo_id, session_id, "A", + decisions=["Use Postgres", "Use Redis", "No auth"], + tasks=["Write schema"]) + + fork_id = _fork(client, repo_id, commit_a["id"]).json()["session_id"] + _send(client, fork_id, "Let's take a different approach.") + + commit_b = _commit(client, repo_id, fork_id, "B", + decisions=["Use MySQL", "Use Redis", "Add JWT auth"], + tasks=["Write schema", "Add migration"]) + + r = client.get(f"/api/v5/lineage/checkpoints/{commit_a['id']}/compare/{commit_b['id']}") + assert r.status_code == 200, r.text + diff = r.json()["diff"] + + assert "Use Postgres" in diff["decisions_only_a"] + assert "No auth" in diff["decisions_only_a"] + assert "Use MySQL" in diff["decisions_only_b"] + assert "Add JWT auth" in diff["decisions_only_b"] + assert "Use Redis" in diff["decisions_shared"] + + assert "Write schema" in diff["tasks_shared"] + assert "Add migration" in diff["tasks_only_b"] + + +def test_compare_nonexistent_checkpoint(client): + """Comparing with a non-existent checkpoint returns 404.""" + repo_id = _create_repo(client, "Compare 404") + session_id = _create_session(client, repo_id) + commit = _commit(client, repo_id, session_id, "real") + fake_id = "00000000-0000-0000-0000-000000000099" + + r = client.get(f"/api/v5/lineage/checkpoints/{commit['id']}/compare/{fake_id}") + assert r.status_code == 404, r.text + + +# ── Regression: existing sessions default to main branch ────────────────────── + +def test_existing_session_defaults_to_main_branch(client): + """Sessions created via the normal (non-fork) path default to branch_name='main'.""" + repo_id = _create_repo(client, "Main Branch Default") + session_id = _create_session(client, repo_id) + + r = client.get(f"/api/v4/chat/sessions/{session_id}") + assert r.status_code == 200, r.text + s = r.json() + assert s["branch_name"] == "main" + assert s["forked_from_checkpoint_id"] is None + + +# ── Branch-semantics regression tests ──────────────────────────────────────── +# These tests cover the real manual-testing bug found in March 2026: +# checkpoints created from forked sessions were silently written to main. + +def test_manual_commit_from_fork_uses_fork_branch(client): + """ + T1: A checkpoint created from a forked session must have branch_name equal + to the fork session's branch, NOT 'main'. + """ + repo_id = _create_repo(client, "BranchSemantics T1") + session_id = _create_session(client, repo_id) + commit_a = _commit(client, repo_id, session_id, "Checkpoint A", + summary="Australia only.", decisions=["Visit Sydney"]) + + fork_resp = _fork(client, repo_id, commit_a["id"], branch_name="au-vietnam") + fork_id = fork_resp.json()["session_id"] + + _send(client, fork_id, "Let's add Vietnam to the trip.") + + fork_commit = _commit(client, repo_id, fork_id, "Fork-local checkpoint", + summary="Australia + Vietnam.", decisions=["Visit Hanoi"]) + + assert fork_commit["branch_name"] == "au-vietnam", ( + f"Expected branch 'au-vietnam', got '{fork_commit['branch_name']}'. " + "manual_commit is still writing fork checkpoints to main." + ) + + +def test_manual_commit_from_fork_first_checkpoint_parents_to_fork_source(client): + """ + T2: The first checkpoint created on a fork branch must have parent_commit_id + equal to the fork source checkpoint, not the main-branch head. + """ + repo_id = _create_repo(client, "BranchSemantics T2") + session_id = _create_session(client, repo_id) + commit_a = _commit(client, repo_id, session_id, "Checkpoint A", + summary="Australia only.") + _send(client, session_id, "Add New Zealand.") + commit_b = _commit(client, repo_id, session_id, "Checkpoint B", + summary="Australia + New Zealand.") # main head is now B + + # Fork from A — not from B + fork_resp = _fork(client, repo_id, commit_a["id"], branch_name="au-only-fork") + fork_id = fork_resp.json()["session_id"] + _send(client, fork_id, "Add Vietnam.") + fork_commit = _commit(client, repo_id, fork_id, "Fork first checkpoint", + summary="Australia + Vietnam.") + + assert fork_commit["parent_commit_id"] == commit_a["id"], ( + f"Expected parent to be checkpoint A ({commit_a['id']}), " + f"got {fork_commit['parent_commit_id']}. " + "Fork's first checkpoint is reparenting to the main-branch head (B) instead of A." + ) + + +def test_manual_commit_from_fork_chains_locally(client): + """ + T3: The second fork-local checkpoint must parent to the first fork-local + checkpoint, not to main-branch head or the fork source again. + """ + repo_id = _create_repo(client, "BranchSemantics T3") + session_id = _create_session(client, repo_id) + commit_a = _commit(client, repo_id, session_id, "A") + + fork_resp = _fork(client, repo_id, commit_a["id"], branch_name="local-chain") + fork_id = fork_resp.json()["session_id"] + + _send(client, fork_id, "Step 1 on fork.") + fork_c1 = _commit(client, repo_id, fork_id, "Fork C1") + + _send(client, fork_id, "Step 2 on fork.") + fork_c2 = _commit(client, repo_id, fork_id, "Fork C2") + + assert fork_c2["parent_commit_id"] == fork_c1["id"], ( + f"Expected Fork C2 to parent Fork C1 ({fork_c1['id']}), " + f"got {fork_c2['parent_commit_id']}. " + "Fork-local checkpoints are not chaining to each other." + ) + + +def test_fork_checkpoint_not_visible_on_main_branch_filter(client): + """ + T4: A checkpoint created on a fork branch must NOT appear when listing + commits filtered to branch='main'. + """ + repo_id = _create_repo(client, "BranchSemantics T4") + session_id = _create_session(client, repo_id) + commit_a = _commit(client, repo_id, session_id, "A on main") + + fork_resp = _fork(client, repo_id, commit_a["id"], branch_name="side-branch") + fork_id = fork_resp.json()["session_id"] + _send(client, fork_id, "Fork-local work.") + fork_commit = _commit(client, repo_id, fork_id, "Fork-local checkpoint") + + main_commits = client.get(f"/api/v2/repos/{repo_id}/commits?branch=main").json() + main_ids = {c["id"] for c in main_commits} + + assert fork_commit["id"] not in main_ids, ( + "Fork-local checkpoint is appearing in the main-branch commit list. " + "It must only appear under its own branch." + ) + + +def test_reachable_checkpoints_main_session_returns_main_only(client): + """ + T5: For a main-branch session the reachable-checkpoints endpoint returns + only main-branch commits. + """ + repo_id = _create_repo(client, "BranchSemantics T5") + session_id = _create_session(client, repo_id) + commit_a = _commit(client, repo_id, session_id, "A") + + fork_resp = _fork(client, repo_id, commit_a["id"], branch_name="other-branch") + fork_id = fork_resp.json()["session_id"] + _send(client, fork_id, "Fork work.") + fork_commit = _commit(client, repo_id, fork_id, "Fork checkpoint") + + r = client.get(f"/api/v5/lineage/sessions/{session_id}/checkpoints") + assert r.status_code == 200, r.text + ids = {c["id"] for c in r.json()} + + assert commit_a["id"] in ids + assert fork_commit["id"] not in ids, ( + "Reachable-checkpoints for a main session is including a fork-branch checkpoint. " + "It should return only main-branch commits." + ) + + +def test_reachable_checkpoints_forked_session_excludes_downstream_main(client): + """ + T6 (NZ regression): The reachable-checkpoints endpoint for a forked session + must NOT include main-branch checkpoints created AFTER the fork point. + + Australia/NZ scenario: + main: A (AU only) → B (AU + NZ) + fork from A: branch 'au-vietnam' + The fork's reachable set must include A but NOT B. + """ + repo_id = _create_repo(client, "BranchSemantics T6") + session_id = _create_session(client, repo_id) + + _send(client, session_id, "Australia trip planning.") + commit_a = _commit(client, repo_id, session_id, "Checkpoint A", + summary="Australia only.", + decisions=["Visit Sydney", "Visit Melbourne"]) + + _send(client, session_id, "Add New Zealand.") + commit_b = _commit(client, repo_id, session_id, "Checkpoint B", + summary="Australia + New Zealand.", + decisions=["Visit Auckland"]) + + # Fork from A + fork_resp = _fork(client, repo_id, commit_a["id"], branch_name="au-vietnam") + fork_id = fork_resp.json()["session_id"] + + r = client.get(f"/api/v5/lineage/sessions/{fork_id}/checkpoints") + assert r.status_code == 200, r.text + reachable = r.json() + ids = {c["id"] for c in reachable} + + assert commit_a["id"] in ids, "Fork source (Checkpoint A) must be reachable." + assert commit_b["id"] not in ids, ( + "Checkpoint B (NZ, created on main AFTER the fork point) must NOT be " + "reachable from the fork session. This is the NZ-bleed regression." + ) + + +def test_reachable_checkpoints_forked_session_includes_fork_local(client): + """ + T7: The reachable-checkpoints endpoint for a forked session includes + checkpoints created on that session's own branch after the fork. + """ + repo_id = _create_repo(client, "BranchSemantics T7") + session_id = _create_session(client, repo_id) + commit_a = _commit(client, repo_id, session_id, "A") + + fork_resp = _fork(client, repo_id, commit_a["id"], branch_name="au-vietnam") + fork_id = fork_resp.json()["session_id"] + _send(client, fork_id, "Vietnam discussion.") + fork_commit = _commit(client, repo_id, fork_id, "Fork checkpoint: AU+VN", + summary="Australia + Vietnam.", decisions=["Visit Hanoi"]) + + r = client.get(f"/api/v5/lineage/sessions/{fork_id}/checkpoints") + assert r.status_code == 200, r.text + ids = {c["id"] for c in r.json()} + + assert fork_commit["id"] in ids, ( + "Fork-local checkpoint must appear in the forked session's reachable set." + ) + assert commit_a["id"] in ids, ( + "Fork source (Checkpoint A) must also appear in the reachable set." + ) + + +def test_draft_isolation_forked_session_no_explicit_mount(client): + """ + T8: draft_checkpoint for a forked session with no explicit mount must + only include fork-local turns (sequence_number > 0). + + We verify this indirectly: the draft endpoint must succeed and not raise, + demonstrating that the session-based isolation path is reached without error. + The turn-boundary correctness is already covered by the send_message + regression test (T6 equivalent in the original test suite). + """ + repo_id = _create_repo(client, "BranchSemantics T8") + session_id = _create_session(client, repo_id) + commit_a = _commit(client, repo_id, session_id, "Checkpoint A", + summary="Australia only.") + + fork_resp = _fork(client, repo_id, commit_a["id"], branch_name="draft-test") + fork_id = fork_resp.json()["session_id"] + _send(client, fork_id, "Let's discuss Vietnam.") + _send(client, fork_id, "We should visit Hanoi and Ho Chi Minh City.") + + # Draft with NO mounted_checkpoint_id — backend must auto-apply fork isolation + r = client.post("/api/v5/checkpoint/draft", json={ + "session_id": fork_id, + "num_turns": 10, + # No mounted_checkpoint_id, no history_base_seq — fork isolation is auto-applied + }) + assert r.status_code == 200, r.text + data = r.json() + # Draft must return a valid (possibly empty) structured object + assert "decisions" in data + assert "tasks" in data + assert "summary" in data diff --git a/backend/tests/unit/__init__.py b/backend/tests/unit/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/backend/tests/unit/test_extractor.py b/backend/tests/unit/test_extractor.py new file mode 100644 index 0000000..26b7d82 --- /dev/null +++ b/backend/tests/unit/test_extractor.py @@ -0,0 +1,165 @@ +"""Unit tests for the extraction service — TDD-first test definitions. + +These tests validate the ExtractorService using the MockProvider, +ensuring schema compliance and structural correctness of extraction results. +""" + +import pytest + +from app.domain.enums import MessageRole +from app.domain.models import ExtractionResult, Message +from app.services.extractor import ExtractorService +from app.services.llm.mock_provider import MockProvider + + +@pytest.fixture +def extractor(): + """ExtractorService with MockProvider.""" + return ExtractorService(llm_provider=MockProvider()) + + +class TestExtractionResultSchema: + """Tests that extraction results conform to expected schema.""" + + @pytest.mark.asyncio + async def test_extraction_result_schema(self, extractor, sample_messages): + """E1: Result matches ExtractionResult structure.""" + result = await extractor.extract(sample_messages) + + assert isinstance(result, ExtractionResult) + assert isinstance(result.summary, str) + assert isinstance(result.decisions, list) + assert isinstance(result.tasks, list) + assert isinstance(result.open_questions, list) + assert isinstance(result.entities, list) + assert isinstance(result.code_snippets, list) + + @pytest.mark.asyncio + async def test_summary_non_empty(self, extractor, sample_messages): + """E2: Summary is non-empty.""" + result = await extractor.extract(sample_messages) + assert len(result.summary) > 0 + + @pytest.mark.asyncio + async def test_summary_bounded_length(self, extractor, sample_messages): + """E3: Summary is under 500 words.""" + result = await extractor.extract(sample_messages) + word_count = len(result.summary.split()) + assert word_count < 500 + + +class TestExtractionArtifactStructure: + """Tests that individual artifact types are well-formed.""" + + @pytest.mark.asyncio + async def test_decisions_well_formed(self, extractor): + """E4: Each Decision has a description field.""" + messages = [ + Message(role=MessageRole.HUMAN, content="We decided to use PostgreSQL", position=0), + Message(role=MessageRole.ASSISTANT, content="Good decision. PostgreSQL is reliable.", position=1), + ] + result = await extractor.extract(messages) + + for decision in result.decisions: + assert hasattr(decision, "description") + assert isinstance(decision.description, str) + assert len(decision.description) > 0 + + @pytest.mark.asyncio + async def test_tasks_well_formed(self, extractor): + """E5: Each Task has description and status.""" + messages = [ + Message(role=MessageRole.HUMAN, content="We need to set up the database. TODO: create schema.", position=0), + Message(role=MessageRole.ASSISTANT, content="I'll add that task.", position=1), + ] + result = await extractor.extract(messages) + + for task in result.tasks: + assert hasattr(task, "description") + assert hasattr(task, "status") + assert isinstance(task.description, str) + assert isinstance(task.status, str) + + @pytest.mark.asyncio + async def test_open_questions_well_formed(self, extractor): + """E6: Each OpenQuestion has a question field.""" + messages = [ + Message(role=MessageRole.HUMAN, content="Should we use Redis? What about caching?", position=0), + Message(role=MessageRole.ASSISTANT, content="Good questions to consider.", position=1), + ] + result = await extractor.extract(messages) + + for question in result.open_questions: + assert hasattr(question, "question") + assert isinstance(question.question, str) + + @pytest.mark.asyncio + async def test_entities_well_formed(self, extractor, sample_messages): + """E7: Each Entity has name and type.""" + result = await extractor.extract(sample_messages) + + assert len(result.entities) > 0 + for entity in result.entities: + assert hasattr(entity, "name") + assert hasattr(entity, "type") + assert isinstance(entity.name, str) + assert isinstance(entity.type, str) + + @pytest.mark.asyncio + async def test_code_snippets_well_formed(self, extractor): + """E8: Each CodeSnippet has language and code.""" + messages = [ + Message( + role=MessageRole.ASSISTANT, + content="Here's the code:\n```python\ndef hello(): pass\n```", + position=0, + ), + ] + result = await extractor.extract(messages) + + for snippet in result.code_snippets: + assert hasattr(snippet, "language") + assert hasattr(snippet, "code") + assert isinstance(snippet.language, str) + assert isinstance(snippet.code, str) + + +class TestExtractionEdgeCases: + """Tests for edge cases in extraction.""" + + @pytest.mark.asyncio + async def test_empty_messages_handling(self, extractor): + """E9: Empty list produces valid but minimal result.""" + result = await extractor.extract([]) + + assert isinstance(result, ExtractionResult) + assert isinstance(result.summary, str) + assert len(result.summary) > 0 # Should have some default message + + @pytest.mark.asyncio + async def test_mock_provider_deterministic(self, extractor, sample_messages): + """E10: Same input produces same output every time.""" + result1 = await extractor.extract(sample_messages) + result2 = await extractor.extract(sample_messages) + + assert result1.summary == result2.summary + assert len(result1.decisions) == len(result2.decisions) + assert len(result1.tasks) == len(result2.tasks) + + @pytest.mark.asyncio + async def test_extraction_with_coding_fixture(self, extractor, coding_transcript): + """E11: Coding transcript fixture extraction succeeds.""" + from app.services.parser import parse_transcript + + messages = parse_transcript(coding_transcript) + result = await extractor.extract(messages) + assert isinstance(result, ExtractionResult) + + @pytest.mark.asyncio + async def test_extraction_with_planning_fixture(self, extractor, planning_transcript): + """E12: Planning transcript fixture extraction succeeds.""" + from app.services.parser import parse_transcript + + messages = parse_transcript(planning_transcript) + result = await extractor.extract(messages) + assert isinstance(result, ExtractionResult) diff --git a/backend/tests/unit/test_golden_outputs.py b/backend/tests/unit/test_golden_outputs.py new file mode 100644 index 0000000..77c0258 --- /dev/null +++ b/backend/tests/unit/test_golden_outputs.py @@ -0,0 +1,189 @@ +"""Golden output tests — snapshot-based validation. + +Verifies that the parser and pack generator produce stable, expected output +for representative fixture transcripts. If a service change alters output, +these tests will catch it. + +To update golden snapshots after intentional changes, run: + pytest tests/unit/test_golden_outputs.py --update-golden +""" + +import json +import pathlib + +import pytest + +from app.domain.enums import MessageRole, TargetTool +from app.domain.models import ( + CodeSnippet, + Decision, + Entity, + ExtractionResult, + OpenQuestion, + Task, +) +from app.services.parser import parse_transcript +from app.services.pack_generator import generate_pack + +FIXTURES_DIR = pathlib.Path(__file__).parent.parent / "fixtures" +GOLDEN_DIR = pathlib.Path(__file__).parent.parent / "golden" + + +def _load_fixture(name: str) -> str: + return (FIXTURES_DIR / name).read_text() + + +def _load_golden(name: str) -> dict | None: + path = GOLDEN_DIR / name + if path.exists(): + return json.loads(path.read_text()) + return None + + +def _save_golden(name: str, data: dict) -> None: + GOLDEN_DIR.mkdir(parents=True, exist_ok=True) + (GOLDEN_DIR / name).write_text(json.dumps(data, indent=2, ensure_ascii=False) + "\n") + + +# ── Standard extraction result used for pack generation goldens ────────── + +STANDARD_EXTRACTION = ExtractionResult( + summary="Session about database setup with SQLAlchemy and Alembic. Covered User and Task models, migration strategy, and input validation with Pydantic.", + decisions=[ + Decision(description="Use SQLAlchemy 2.0 mapped_column style", context="Modern approach"), + Decision(description="Use Alembic with autogenerate for migrations", context="Standard SQLAlchemy migration tool"), + Decision(description="No soft delete for MVP", context="Keep it simple"), + ], + tasks=[ + Task(description="Add indexes on email and status columns", status="pending"), + Task(description="Create initial migration", status="pending"), + Task(description="Add error handling middleware", status="pending"), + ], + open_questions=[ + OpenQuestion(question="Should we add pagination to the task listing endpoint?", context="API design"), + ], + entities=[ + Entity(name="SQLAlchemy", type="technology", context="ORM framework"), + Entity(name="Alembic", type="technology", context="Migration tool"), + Entity(name="Pydantic", type="technology", context="Validation"), + Entity(name="models.py", type="file", context="Database models"), + ], + code_snippets=[ + CodeSnippet( + language="python", + code="class User(Base):\n __tablename__ = 'users'\n id: Mapped[int] = mapped_column(primary_key=True)", + description="User model", + ), + ], +) + + +class TestParserGolden: + """Verify parser output stability for fixture transcripts.""" + + @pytest.mark.parametrize( + "fixture_name", + ["coding_session.txt", "planning_session.txt", "code_heavy_session.txt", "unlabeled_session.txt"], + ) + def test_parser_output_stable(self, fixture_name, request): + """Parser produces consistent output for fixture transcripts.""" + raw = _load_fixture(fixture_name) + messages = parse_transcript(raw) + + golden_name = f"parser_{fixture_name.replace('.txt', '.json')}" + actual = { + "message_count": len(messages), + "roles": [m.role.value for m in messages], + "positions": [m.position for m in messages], + "content_lengths": [len(m.content) for m in messages], + } + + golden = _load_golden(golden_name) + if golden is None or request.config.getoption("--update-golden", default=False): + _save_golden(golden_name, actual) + pytest.skip(f"Golden snapshot created: {golden_name}") + else: + assert actual["message_count"] == golden["message_count"], ( + f"Message count changed: {golden['message_count']} → {actual['message_count']}" + ) + assert actual["roles"] == golden["roles"], "Role detection changed" + assert actual["positions"] == golden["positions"], "Positions changed" + + def test_coding_session_structure(self): + """Coding fixture produces expected message structure.""" + raw = _load_fixture("coding_session.txt") + messages = parse_transcript(raw) + + assert len(messages) >= 4 # Multi-turn conversation + assert messages[0].role == MessageRole.HUMAN + assert messages[1].role == MessageRole.ASSISTANT + # Should preserve code blocks + has_code = any("```" in m.content for m in messages) + assert has_code, "Code blocks should be preserved in messages" + + def test_unlabeled_session_fallback(self): + """Unlabeled fixture falls back to single unknown message.""" + raw = _load_fixture("unlabeled_session.txt") + messages = parse_transcript(raw) + + assert len(messages) == 1 + assert messages[0].role == MessageRole.UNKNOWN + + +class TestPackGeneratorGolden: + """Verify pack generator output stability.""" + + @pytest.mark.parametrize("target", list(TargetTool)) + def test_pack_output_stable(self, target, request): + """Pack generator produces consistent output for standard extraction.""" + pack = generate_pack(STANDARD_EXTRACTION, target) + + golden_name = f"pack_{target.value}.json" + actual = { + "target_tool": pack.target_tool.value, + "format": pack.format, + "content_length": len(pack.content), + "content": pack.content, + } + + golden = _load_golden(golden_name) + if golden is None or request.config.getoption("--update-golden", default=False): + _save_golden(golden_name, actual) + pytest.skip(f"Golden snapshot created: {golden_name}") + else: + assert actual["content"] == golden["content"], ( + f"Pack content changed for target {target.value}" + ) + + def test_chatgpt_pack_structure(self): + """ChatGPT pack has expected conversational structure.""" + pack = generate_pack(STANDARD_EXTRACTION, TargetTool.CHATGPT) + assert "I'm continuing" in pack.content + assert "Key decisions already made" in pack.content + assert "Outstanding tasks" in pack.content + + def test_claude_pack_structure(self): + """Claude pack has expected XML structure.""" + pack = generate_pack(STANDARD_EXTRACTION, TargetTool.CLAUDE) + assert "" in pack.content + assert "" in pack.content + assert "" in pack.content + assert "" in pack.content + + def test_cursor_pack_structure(self): + """Cursor pack has expected code-focused structure.""" + pack = generate_pack(STANDARD_EXTRACTION, TargetTool.CURSOR) + assert "# Continuation Context" in pack.content + assert "## Task Checklist" in pack.content + assert "[ ]" in pack.content # Pending tasks + assert "## Technologies" in pack.content + + def test_generic_pack_structure(self): + """Generic pack has all expected Markdown sections.""" + pack = generate_pack(STANDARD_EXTRACTION, TargetTool.GENERIC) + assert "# Session Continuation Pack" in pack.content + assert "## Summary" in pack.content + assert "## Key Decisions" in pack.content + assert "## Tasks" in pack.content + assert "## Open Questions" in pack.content + assert "## Entities" in pack.content diff --git a/backend/tests/unit/test_pack_generator.py b/backend/tests/unit/test_pack_generator.py new file mode 100644 index 0000000..caa90dc --- /dev/null +++ b/backend/tests/unit/test_pack_generator.py @@ -0,0 +1,152 @@ +"""Unit tests for context pack generation — TDD-first test definitions. + +These tests validate the generate_pack() pure function which renders +target-specific continuation packs from extraction results. +""" + +import pytest + +from app.domain.enums import TargetTool +from app.domain.models import ExtractionResult +from app.services.pack_generator import generate_pack + + +class TestChatGPTPack: + """Tests for ChatGPT-targeted pack generation.""" + + def test_chatgpt_pack_format(self, sample_extraction_result): + """G1: ChatGPT pack contains conversational continuation prompt.""" + pack = generate_pack(sample_extraction_result, TargetTool.CHATGPT) + + assert "continuing" in pack.content.lower() or "continue" in pack.content.lower() + assert pack.target_tool == TargetTool.CHATGPT + + def test_chatgpt_pack_includes_summary(self, sample_extraction_result): + """G5 (ChatGPT): Summary is present in pack.""" + pack = generate_pack(sample_extraction_result, TargetTool.CHATGPT) + # Summary content should appear + assert "SQLAlchemy" in pack.content or "database" in pack.content.lower() + + +class TestClaudePack: + """Tests for Claude-targeted pack generation.""" + + def test_claude_pack_format(self, sample_extraction_result): + """G2: Claude pack contains structured XML-style sections.""" + pack = generate_pack(sample_extraction_result, TargetTool.CLAUDE) + + assert "" in pack.content + assert "" in pack.content + assert "" in pack.content + assert pack.target_tool == TargetTool.CLAUDE + + def test_claude_pack_includes_decisions(self, sample_extraction_result): + """G7 (Claude): Decisions are included in pack.""" + pack = generate_pack(sample_extraction_result, TargetTool.CLAUDE) + assert "" in pack.content + assert "PostgreSQL" in pack.content + + +class TestCursorPack: + """Tests for Cursor-targeted pack generation.""" + + def test_cursor_pack_format(self, sample_extraction_result): + """G3: Cursor pack contains code-focused content with task list.""" + pack = generate_pack(sample_extraction_result, TargetTool.CURSOR) + + assert "## Task Checklist" in pack.content or "## Tasks" in pack.content + assert "[ ]" in pack.content or "[x]" in pack.content + assert pack.target_tool == TargetTool.CURSOR + + def test_cursor_pack_code_context(self, sample_extraction_result): + """G3b: Cursor pack includes code context.""" + pack = generate_pack(sample_extraction_result, TargetTool.CURSOR) + assert "## Code Context" in pack.content or "```" in pack.content + + +class TestGenericPack: + """Tests for generic Markdown pack generation.""" + + def test_generic_pack_format(self, sample_extraction_result): + """G4: Generic pack is clean Markdown with all sections.""" + pack = generate_pack(sample_extraction_result, TargetTool.GENERIC) + + assert "# Session Continuation Pack" in pack.content + assert "## Summary" in pack.content + assert pack.target_tool == TargetTool.GENERIC + assert pack.format == "markdown" + + def test_generic_pack_includes_all_sections(self, sample_extraction_result): + """G4b: Generic pack includes all artifact sections.""" + pack = generate_pack(sample_extraction_result, TargetTool.GENERIC) + + assert "## Key Decisions" in pack.content + assert "## Tasks" in pack.content + assert "## Open Questions" in pack.content + assert "## Entities" in pack.content + + +class TestPackContent: + """Tests for pack content quality across all targets.""" + + @pytest.mark.parametrize("target", list(TargetTool)) + def test_pack_includes_summary(self, sample_extraction_result, target): + """G5: Summary present in every target format.""" + pack = generate_pack(sample_extraction_result, target) + # The summary content or a reference to it should appear + assert len(pack.content) > 50 # Pack should have meaningful content + + @pytest.mark.parametrize("target", list(TargetTool)) + def test_pack_includes_tasks(self, sample_extraction_result, target): + """G6: Active tasks listed in pack.""" + pack = generate_pack(sample_extraction_result, target) + # Tasks should be mentioned + assert "migration" in pack.content.lower() or "task" in pack.content.lower() + + @pytest.mark.parametrize("target", list(TargetTool)) + def test_pack_includes_open_questions(self, sample_extraction_result, target): + """G8: Open questions included in pack.""" + pack = generate_pack(sample_extraction_result, target) + assert "async" in pack.content.lower() or "question" in pack.content.lower() + + @pytest.mark.parametrize("target", list(TargetTool)) + def test_pack_concise(self, sample_extraction_result, target): + """G9: Output under 2000 words.""" + pack = generate_pack(sample_extraction_result, target) + word_count = len(pack.content.split()) + assert word_count < 2000 + + def test_all_targets_produce_different_output(self, sample_extraction_result): + """G11: Each target format is distinct.""" + packs = { + target: generate_pack(sample_extraction_result, target) + for target in TargetTool + } + + contents = [p.content for p in packs.values()] + # All should be unique + assert len(set(contents)) == len(contents) + + +class TestPackEdgeCases: + """Edge case tests for pack generation.""" + + @pytest.mark.parametrize("target", list(TargetTool)) + def test_empty_extraction_result(self, target): + """G10: Empty extraction result produces valid but minimal pack.""" + empty_result = ExtractionResult(summary="No significant content found.") + pack = generate_pack(empty_result, target) + + assert isinstance(pack.content, str) + assert len(pack.content) > 0 + assert pack.target_tool == target + + @pytest.mark.parametrize("target", list(TargetTool)) + def test_pack_is_copy_paste_ready(self, sample_extraction_result, target): + """G12: No broken formatting in output.""" + pack = generate_pack(sample_extraction_result, target) + + # Should not have incomplete markdown or broken tags + assert pack.content.count("```") % 2 == 0 # Even number of code fences + if target == TargetTool.CLAUDE: + assert pack.content.count("") == pack.content.count("") diff --git a/backend/tests/unit/test_parser.py b/backend/tests/unit/test_parser.py new file mode 100644 index 0000000..5c5d4a7 --- /dev/null +++ b/backend/tests/unit/test_parser.py @@ -0,0 +1,452 @@ +"""Unit tests for transcript parser — TDD-first test definitions. + +These tests validate the pure parse_transcript() function which converts +raw transcript text into structured Message objects. +""" + +import pytest + +from app.domain.enums import MessageRole +from app.services.parser import parse_transcript + + +class TestParseRoleLabeled: + """Tests for transcripts with explicit role labels.""" + + def test_parse_role_labeled_simple(self): + """P1: Simple Human/Assistant transcript produces 2 messages.""" + raw = "Human: Hi there\nAssistant: Hello! How can I help?" + messages = parse_transcript(raw) + + assert len(messages) == 2 + assert messages[0].role == MessageRole.HUMAN + assert messages[0].content == "Hi there" + assert messages[1].role == MessageRole.ASSISTANT + assert messages[1].content == "Hello! How can I help?" + + def test_parse_role_labeled_multiline(self): + """P2: Multi-line messages preserve full content including newlines.""" + raw = "Human: I have a question.\nIt's about databases.\nAssistant: Sure, I can help.\nWhat would you like to know?" + messages = parse_transcript(raw) + + assert len(messages) == 2 + assert "question" in messages[0].content + assert "databases" in messages[0].content + assert "help" in messages[1].content + + def test_parse_multi_turn(self): + """P3: 5+ turn realistic transcript parses all turns correctly.""" + raw = ( + "Human: Help me with Python\n" + "Assistant: Sure! What do you need?\n" + "Human: How do I read a file?\n" + "Assistant: Use the open() function.\n" + "Human: Can you show me an example?\n" + "Assistant: Here you go: with open('file.txt') as f: data = f.read()" + ) + messages = parse_transcript(raw) + + assert len(messages) == 6 + assert all(m.role in (MessageRole.HUMAN, MessageRole.ASSISTANT) for m in messages) + # Check alternating roles + for i in range(len(messages)): + expected = MessageRole.HUMAN if i % 2 == 0 else MessageRole.ASSISTANT + assert messages[i].role == expected + + def test_parse_message_positions(self): + """P13: Position field increments correctly across messages.""" + raw = "Human: First\nAssistant: Second\nHuman: Third" + messages = parse_transcript(raw) + + assert len(messages) == 3 + assert messages[0].position == 0 + assert messages[1].position == 1 + assert messages[2].position == 2 + + +class TestParseUnlabeled: + """Tests for transcripts without role labels.""" + + def test_parse_unlabeled_transcript(self): + """P4: Plain text without role markers returns single unknown message.""" + raw = "This is just a paragraph of text about databases and APIs." + messages = parse_transcript(raw) + + assert len(messages) == 1 + assert messages[0].role == MessageRole.UNKNOWN + assert "databases" in messages[0].content + + def test_parse_empty_transcript(self): + """P7: Empty string returns empty list.""" + messages = parse_transcript("") + assert messages == [] + + def test_parse_whitespace_only(self): + """P8: Whitespace-only transcript returns empty list.""" + messages = parse_transcript(" \n\n ") + assert messages == [] + + +class TestParseCodeBlocks: + """Tests for code block preservation.""" + + def test_parse_code_blocks_preserved(self): + """P5: Code blocks within messages are preserved intact.""" + raw = "Human: Can you show me a function?\nAssistant: Here's a function:\n```python\ndef hello():\n print('hello')\n```\nThat should work." + messages = parse_transcript(raw) + + assert len(messages) == 2 + # Code block should be in assistant's message + assert "```python" in messages[1].content + assert "def hello():" in messages[1].content + assert "```" in messages[1].content + + +class TestParseAlternativeMarkers: + """Tests for alternative role marker formats.""" + + def test_parse_alternative_markers_user_ai(self): + """P9: User/AI markers are recognized.""" + raw = "User: Hello\nAI: Hi there!" + messages = parse_transcript(raw) + + assert len(messages) == 2 + assert messages[0].role == MessageRole.HUMAN + assert messages[1].role == MessageRole.ASSISTANT + + def test_parse_alternative_markers_you(self): + """P9b: 'You:' marker recognized as human.""" + raw = "You: What's the weather?\nAssistant: I can't check weather." + messages = parse_transcript(raw) + + assert len(messages) == 2 + assert messages[0].role == MessageRole.HUMAN + + def test_parse_mixed_markers(self): + """P10: Mixed marker formats are handled.""" + raw = "Human: Hi\nChatGPT: Hello!" + messages = parse_transcript(raw) + + assert len(messages) == 2 + assert messages[0].role == MessageRole.HUMAN + assert messages[1].role == MessageRole.ASSISTANT + + def test_parse_colon_in_content(self): + """P14: Content containing colons (like URLs) is preserved.""" + raw = "Human: Check out http://example.com for more info" + messages = parse_transcript(raw) + + assert len(messages) == 1 + assert "http://example.com" in messages[0].content + + +class TestParseEdgeCases: + """Edge case tests for the parser.""" + + def test_parse_special_characters(self): + """P12: Special characters and unicode are handled.""" + raw = "Human: What about émojis 🎉 and spëcial chars?\nAssistant: They work fine! 中文也可以" + messages = parse_transcript(raw) + + assert len(messages) == 2 + assert "🎉" in messages[0].content + assert "中文" in messages[1].content + + def test_parse_long_transcript(self): + """P11: Very long transcripts are parsed completely.""" + # Generate a long transcript + turns = [] + for i in range(100): + turns.append(f"Human: Message {i} " + "word " * 50) + turns.append(f"Assistant: Response {i} " + "word " * 50) + raw = "\n".join(turns) + + messages = parse_transcript(raw) + assert len(messages) == 200 + + def test_parse_with_fixture_coding(self, coding_transcript): + """Coding fixture parses without error.""" + messages = parse_transcript(coding_transcript) + assert len(messages) >= 1 + + def test_parse_with_fixture_planning(self, planning_transcript): + """Planning fixture parses without error.""" + messages = parse_transcript(planning_transcript) + assert len(messages) >= 1 + + def test_parse_with_fixture_unlabeled(self, unlabeled_transcript): + """Unlabeled fixture parses without crashing.""" + messages = parse_transcript(unlabeled_transcript) + assert len(messages) >= 1 + + def test_parse_with_fixture_code_heavy(self, code_heavy_transcript): + """Code-heavy fixture parses without error.""" + messages = parse_transcript(code_heavy_transcript) + assert len(messages) >= 1 + + +# ═════════════════════════════════════════════════════════════════════════ +# NEW TEST CASES — Real-world transcript formats +# ═════════════════════════════════════════════════════════════════════════ + + +class TestChatGPTWebCopyPaste: + """Tests for ChatGPT web UI copy-paste format ('You said:' / 'ChatGPT said:').""" + + def test_chatgpt_web_basic(self): + """N1: Basic ChatGPT web copy-paste with 'You said:' / 'ChatGPT said:'.""" + raw = ( + "You said:\n" + "Help me with Python decorators\n\n" + "ChatGPT said:\n" + "Decorators are functions that modify the behavior of other functions." + ) + messages = parse_transcript(raw) + + assert len(messages) == 2 + assert messages[0].role == MessageRole.HUMAN + assert "decorators" in messages[0].content + assert messages[1].role == MessageRole.ASSISTANT + assert "modify" in messages[1].content + + def test_chatgpt_web_multi_turn(self): + """N2: Multi-turn ChatGPT web conversation.""" + raw = ( + "You said:\n" + "Can you help me refactor the auth module?\n\n" + "ChatGPT said:\n" + "Sure! JWT is a great choice for REST APIs.\n\n" + "You said:\n" + "What about token rotation?\n\n" + "ChatGPT said:\n" + "Refresh token rotation is a security best practice." + ) + messages = parse_transcript(raw) + + assert len(messages) == 4 + assert messages[0].role == MessageRole.HUMAN + assert messages[1].role == MessageRole.ASSISTANT + assert messages[2].role == MessageRole.HUMAN + assert messages[3].role == MessageRole.ASSISTANT + + def test_chatgpt_web_with_code_blocks(self): + """N3: ChatGPT web format preserves code blocks.""" + raw = ( + "You said:\n" + "Show me a retry decorator\n\n" + "ChatGPT said:\n" + "Here's a retry decorator:\n\n" + "```python\n" + "def retry(func):\n" + " def wrapper(*args):\n" + " return func(*args)\n" + " return wrapper\n" + "```" + ) + messages = parse_transcript(raw) + + assert len(messages) == 2 + assert "```python" in messages[1].content + assert "def retry" in messages[1].content + + +class TestChatGPTSharedLink: + """Tests for ChatGPT shared link format (bare 'User' / 'Assistant' on own line).""" + + def test_shared_link_basic(self): + """N4: ChatGPT shared link format with bare role names.""" + raw = ( + "User\n" + "I need help designing a database schema.\n\n" + "Assistant\n" + "Here's a normalized schema design with Users and Orders tables." + ) + messages = parse_transcript(raw) + + assert len(messages) == 2 + assert messages[0].role == MessageRole.HUMAN + assert "database schema" in messages[0].content + assert messages[1].role == MessageRole.ASSISTANT + + def test_shared_link_multi_turn(self): + """N5: Multi-turn shared link format.""" + raw = ( + "User\n" + "How do I set up migrations?\n\n" + "Assistant\n" + "Use Alembic with autogenerate.\n\n" + "User\n" + "Should I denormalize anything?\n\n" + "Assistant\n" + "Consider denormalizing for read-heavy operations." + ) + messages = parse_transcript(raw) + + assert len(messages) == 4 + assert messages[0].role == MessageRole.HUMAN + assert messages[3].role == MessageRole.ASSISTANT + + +class TestMarkdownBoldFormat: + """Tests for markdown bold role markers (**User**: / **Assistant**:).""" + + def test_markdown_bold_basic(self): + """N6: Basic markdown bold format.""" + raw = ( + "**User**: I want to add error handling to the API.\n\n" + "**Assistant**: Here's a global exception handler for FastAPI." + ) + messages = parse_transcript(raw) + + assert len(messages) == 2 + assert messages[0].role == MessageRole.HUMAN + assert messages[1].role == MessageRole.ASSISTANT + + def test_markdown_bold_with_code(self): + """N7: Markdown bold format with code blocks.""" + raw = ( + "**User**: Show me a FastAPI route\n\n" + "**Assistant**: Here you go:\n\n" + "```python\n" + "@app.get('/items')\n" + "def get_items():\n" + " return []\n" + "```\n\n" + "**User**: Can you add query params?\n\n" + "**Assistant**: Add a `skip: int = 0` parameter." + ) + messages = parse_transcript(raw) + + assert len(messages) == 4 + assert "```python" in messages[1].content + assert messages[2].role == MessageRole.HUMAN + assert messages[3].role == MessageRole.ASSISTANT + + def test_markdown_bold_human_variant(self): + """N8: Markdown bold with 'Human' instead of 'User'.""" + raw = ( + "**Human**: What's the best ORM for Python?\n\n" + "**Assistant**: SQLAlchemy is the most popular choice." + ) + messages = parse_transcript(raw) + + assert len(messages) == 2 + assert messages[0].role == MessageRole.HUMAN + + +class TestAngleBracketFormat: + """Tests for angle-bracket format ( / ).""" + + def test_angle_bracket_basic(self): + """N9: Angle bracket format.""" + raw = ( + "\n" + "How do I deploy to AWS?\n\n" + "\n" + "I recommend using AWS ECS with Docker." + ) + messages = parse_transcript(raw) + + assert len(messages) == 2 + assert messages[0].role == MessageRole.HUMAN + assert messages[1].role == MessageRole.ASSISTANT + + def test_angle_bracket_multi_turn(self): + """N10: Multi-turn angle bracket format.""" + raw = ( + "\n" + "What's the difference between REST and GraphQL?\n\n" + "\n" + "REST uses multiple endpoints, GraphQL uses a single endpoint.\n\n" + "\n" + "Which should I use?\n\n" + "\n" + "It depends on your use case." + ) + messages = parse_transcript(raw) + + assert len(messages) == 4 + + +class TestMixedAndEdgeCases: + """Mixed format and edge case tests.""" + + def test_transcript_with_urls_not_confused(self): + """N11: URLs containing role words don't trigger false splits.""" + raw = "Human: Check https://assistant.google.com for more info\nAssistant: That's a useful link!" + messages = parse_transcript(raw) + + assert len(messages) == 2 + assert "https://assistant.google.com" in messages[0].content + + def test_case_insensitive_markers(self): + """N12: Markers are case-insensitive.""" + raw = "HUMAN: Hello\nASSISTANT: Hi there!" + messages = parse_transcript(raw) + + assert len(messages) == 2 + assert messages[0].role == MessageRole.HUMAN + assert messages[1].role == MessageRole.ASSISTANT + + def test_extra_whitespace_between_messages(self): + """N13: Extra blank lines between messages are handled.""" + raw = ( + "Human: First question\n\n\n\n" + "Assistant: First answer\n\n\n" + "Human: Second question\n\n" + "Assistant: Second answer" + ) + messages = parse_transcript(raw) + + assert len(messages) == 4 + + def test_very_long_messages(self): + """N14: Very long individual messages are preserved.""" + long_content = "word " * 500 + raw = f"Human: {long_content}\nAssistant: Got it." + messages = parse_transcript(raw) + + assert len(messages) == 2 + assert len(messages[0].content) > 2000 + + def test_code_block_with_human_keyword(self): + """N15: Code blocks containing 'Human:' don't cause false splits.""" + raw = ( + "Human: How do I parse this?\n" + "Assistant: Here's how:\n" + "```\n" + "# This line says Human: something\n" + "data = parse(input)\n" + "```\n" + "That should work." + ) + messages = parse_transcript(raw) + + # Should produce 2 messages — the code block stays in assistant's message + assert len(messages) >= 2 + assert messages[0].role == MessageRole.HUMAN + + def test_single_role_marker(self): + """N16: Single role marker with no alternation produces one message.""" + raw = "Human: Just a single question with no response yet." + messages = parse_transcript(raw) + + assert len(messages) == 1 + assert messages[0].role == MessageRole.HUMAN + + def test_copilot_marker(self): + """N17: 'Copilot:' marker recognized as assistant.""" + raw = "User: Help me\nCopilot: Sure thing!" + messages = parse_transcript(raw) + + assert len(messages) == 2 + assert messages[1].role == MessageRole.ASSISTANT + + def test_gpt4o_marker(self): + """N18: 'GPT-4o:' variant recognized.""" + raw = "User: What's new?\nGPT-4o: I have improved capabilities." + messages = parse_transcript(raw) + + assert len(messages) == 2 + assert messages[1].role == MessageRole.ASSISTANT + diff --git a/demos/branching-reasoning-demo/DEMO_SCRIPT.md b/demos/branching-reasoning-demo/DEMO_SCRIPT.md new file mode 100644 index 0000000..64e4954 --- /dev/null +++ b/demos/branching-reasoning-demo/DEMO_SCRIPT.md @@ -0,0 +1,67 @@ +# Demo Script: Exact Messages + +Paste these messages verbatim during the demo. They are written to produce consistent, structured AI responses that look good in the checkpoint diff. + +If you are using **Mock Mode**, responses will be deterministic simulated outputs. If you are using a real provider, responses will vary but the structure of the conversation will still work. + +--- + +## Section 1: Main Branch Thread (Retrieval-Heavy / Enterprise) + +### Turn 1 — Scope setting + +``` +We're designing an autonomous research agent for an enterprise knowledge management platform. The agent needs to answer deep analytical questions by searching across a corpus of internal documents, research papers, and structured data. What are the key architectural decisions we need to make upfront? +``` + +### Turn 2 — Retrieval-layer constraints + +``` +Our corpus is large — roughly 2 million documents, updated daily. We need sub-second retrieval latency even for complex multi-hop queries. What retrieval architecture should we use, and how does it affect our indexing pipeline design? +``` + +### Turn 3 — Enterprise deployment assumptions + +``` +The platform is deployed on-premises with strict data residency requirements. We can't use external embedding APIs — all models need to run locally or in our private cloud. We also need audit trails for every retrieval decision. How does this constrain the design? +``` + +### Turn 4 — Decision point: commit to RAG with vector index + reranker + +``` +Based on what we've discussed: we're going with a two-stage retrieval pipeline — a vector index for coarse candidate retrieval, followed by a cross-encoder reranker for precision. Tool calls from the agent are gated by retrieval confidence scores. Does this architecture have any critical failure modes we should document now? +``` + +**→ After this turn, create Checkpoint A on main branch.** + +--- + +## Section 2: Fork Branch Thread (State-First / Individual Researcher) + +*(Fork from Checkpoint A before starting this section)* + +### Fork Turn 1 — Challenge the retrieval-first assumption + +``` +I want to step back and challenge the retrieval-first assumption we just committed to on the main branch. For an individual researcher working with a smaller, more curated corpus — maybe 10,000 papers in their domain — a heavy vector indexing pipeline seems like overkill. What's the alternative architectural paradigm? +``` + +### Fork Turn 2 — Advocate for explicit reasoning state + +``` +The thing I keep coming back to is that the most valuable part of a research agent isn't fast retrieval — it's the quality of the reasoning chain. If we make the agent's reasoning state first-class — explicit, persistent, and inspectable — we get better science even if retrieval is slower. What does an architecture built around reasoning state look like? +``` + +### Fork Turn 3 — Propose scratchpad + plan/act loop + +``` +I'm thinking of a scratchpad architecture: the agent maintains a persistent reasoning scratchpad across turns, plans its next retrieval or synthesis step explicitly, acts, then updates the scratchpad. The corpus can be a simple file system or SQLite — no vector index needed for this scale. What does the plan/act loop look like, and what are the tradeoffs vs the RAG approach? +``` + +### Fork Turn 4 — Address individual researcher use case + +``` +For an individual researcher, the key requirements are: runs locally on a laptop, no external services, reasoning is transparent and auditable, can handle 10k-100k documents without a dedicated index server. Summarise the final architecture for this use case and list the decisions we've made that differ from the enterprise RAG approach. +``` + +**→ After this turn, create Checkpoint B on the state-first-reasoning branch.** diff --git a/demos/branching-reasoning-demo/EXPECTED_OUTCOMES.md b/demos/branching-reasoning-demo/EXPECTED_OUTCOMES.md new file mode 100644 index 0000000..fe0079c --- /dev/null +++ b/demos/branching-reasoning-demo/EXPECTED_OUTCOMES.md @@ -0,0 +1,94 @@ +# Expected Outcomes + +Use this as a checklist when running or validating the demo. If any outcome is missing or wrong, re-check the corresponding step in RUNBOOK.md. + +--- + +## Checkpoint A (main branch — retrieval-heavy) + +**Branch:** `main` + +**Objective:** Build an enterprise-grade autonomous research agent with sub-second retrieval over a 2M-document corpus, deployed on-premises with data residency and audit requirements. + +**Expected decisions (at least these, possibly more):** +- Two-stage retrieval: vector index (coarse) → cross-encoder reranker (precision) +- All embedding and reranking models run locally / private cloud +- Agent tool calls gated by retrieval confidence score threshold +- Audit trail for every retrieval decision (required by enterprise compliance) +- Indexing pipeline handles daily incremental updates at scale + +**Expected open questions:** +- What confidence threshold to use for gating tool calls? +- How to handle retrieval failures gracefully (fallback strategy)? +- Cross-encoder reranker latency under peak load? + +--- + +## Checkpoint B (state-first-reasoning branch) + +**Branch:** `state-first-reasoning` + +**Objective:** Build a lightweight autonomous research agent for individual researchers: local deployment, transparent reasoning, no external services, handles 10k–100k documents without a dedicated index server. + +**Expected decisions (at least these, possibly more):** +- Scratchpad-based reasoning: persistent, explicit reasoning state across turns +- Plan/act loop: agent explicitly plans next step, acts, then updates scratchpad +- No vector index: file system or SQLite for document storage at this scale +- No external API dependencies — fully local +- Reasoning chain is first-class, inspectable, and auditable by the researcher + +**Expected open questions:** +- How to structure the scratchpad for long multi-day research sessions? +- When does the corpus outgrow SQLite and require an upgrade path? +- How to handle conflicting evidence across documents in the scratchpad? + +--- + +## Checkpoint comparison diff + +When comparing Checkpoint A vs Checkpoint B, the diff should show: + +### Decisions only in A (retrieval-heavy): +- Vector index for coarse candidate retrieval +- Cross-encoder reranker for precision retrieval +- Retrieval confidence score gating for tool calls +- On-premises / private cloud model deployment +- Audit trail per retrieval decision + +### Decisions only in B (state-first): +- Scratchpad as first-class reasoning state +- Plan/act loop with explicit next-step planning +- No vector index — SQLite or file system storage +- Fully local, no external service dependencies +- Researcher-auditable reasoning chain + +### Decisions shared by both: +- Agent autonomy goal (multi-hop reasoning over document corpus) +- Tool-call interface pattern (agent decides when to retrieve) +- Transparency/auditability requirement (different mechanism, shared goal) + +--- + +## UI checkpoints + +| What to verify | Expected state | +|---|---| +| Checkpoint A branch field | `main` | +| Checkpoint B branch field | `state-first-reasoning` | +| Fork session empty state | "Exploring alternate direction — Forked from \" | +| Checkpoint history panel in fork session | Shows Checkpoint A + ancestors only (not downstream main commits) | +| Compose bar mock toggle label | "Mock Mode" / "Mock Mode Active" (not "Demo Mode") | +| Branch chip in fork session | Shows `state-first-reasoning`, clickable, opens history panel | +| FORKED chip | Clickable, shows Checkpoint A hash, opens history panel on click | + +--- + +## What a broken demo looks like + +| Symptom | Likely cause | +|---|---| +| Checkpoint B appears on `main` branch | Session `branch_name` not set correctly on fork creation | +| Fork checkpoint panel shows main-branch downstream commits | Reachable checkpoint query not using branch-local semantics | +| Draft for Checkpoint B mentions enterprise/RAG decisions | Draft not isolating fork turns (fetching all turns instead of fork-local) | +| FORKED chip is not clickable | Frontend still rendering FORKED as `` not ` + + + {/* Scrollable body */} +
+ {error && ( +
+ {error} +
+ )} + + {/* Commit message */} + + setMessage(e.target.value)} + placeholder="e.g. Middleware wired, DB schema finalized" + className="input-field" + required + disabled={loading} + /> + + + {/* Author */} +
+ + setAuthorAgent(e.target.value)} + placeholder="claude, cursor, user…" + className="input-field" + disabled={loading} + /> + + + + +
+ + {/* Objective */} + + setObjective(e.target.value)} + placeholder="e.g. Complete auth middleware setup" + className="input-field" + disabled={loading} + /> + + + {/* Summary */} + +