# DEPRECATED - Forgejo + Woodpecker VPS A-to-Z Runbook (Sensitive) - VPS 12-24-720 Ubuntu 24.04 HyperTwist migration copy

Last updated: 2026-05-29 (Europe/Berlin)
Scope: Derived sensitive migration copy of the current Forgejo/Woodpecker VPS runbook, extended for the new IONOS `VPS 12-24-720` Ubuntu `24.04` Linux target and the HyperTwist migration/cutover sequence.
Classification: **Highly Sensitive** (contains real credentials, keys/paths, and secrets).

Derived-from note:
- source copy: `C:\HyperTwist\docs\refs\FORGEJO_WOODPECKER_VPS_A_TO_Z_SENSITIVE_RUNBOOK.md`
- this file keeps the original live runbook content intact and appends the new-server migration plan, pending operator inputs, and remote-development setup instructions

## 1. Security Notice

This file intentionally stores live and historical credentials because that was explicitly requested.

Immediate hardening recommended after successful handover:
- Rotate root/deploy OS passwords.
- Rotate Forgejo admin password.
- Rotate DB password and Woodpecker OAuth/agent secrets.
- Move all secrets to a dedicated vault.
- Keep this file out of any public repository.

## 2. Environment Inventory

- Provider: IONOS
- VPS IP: `212.227.22.66`
- Preferred server/display name in ops and pgAdmin: `vps-forgejo-cicd` (label only, not an OS login user)
- OS: Ubuntu `24.04.4 LTS`
- Domains:
  - `git.scriptoriumai.io`
  - `ci.scriptoriumai.io`
- Stack root on VPS: `/srv/scriptorium-stack`
- Local project root: `C:\ScriptoriumAI`

## 3. Credential Ledger (Current + Historical)

## 3.1 VM / SSH / Console

- Host: `212.227.22.66`
- Root user: `root`
- IONOS initial password: intentionally removed from this document after credential exchange.
- Recovery password used during rescue flow (historical): `temproot2026`
- Bootstrap-generated OS password in local artifact:
  - File: `C:\ScriptoriumAI\ops\ssh\vps_generated_password.txt`
  - Value: `VjU!pp7jQc@ZdQzb+uCP6#C6EZSi!8`

## 3.2 SSH Key Material

- Private key: `C:\ScriptoriumAI\ops\ssh\scriptorium_vps_ed25519`
- Public key: `C:\ScriptoriumAI\ops\ssh\scriptorium_vps_ed25519.pub`
- Pubkey value used by bootstrap script:
  - `ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAnFyOdMl7NxjYanwr/dF6Rx8iWEzrv1zojQZHuQuEi/ scriptorium-vps-2026-02-28`

## 3.3 Application and DB Secrets (Current)

Source of truth: `/srv/scriptorium-stack/.env`

- `DOMAIN_GIT=git.scriptoriumai.io`
- `DOMAIN_CI=ci.scriptoriumai.io`
- `POSTGRES_DB=forgejo`
- `POSTGRES_USER=forgejo`
- `POSTGRES_PASSWORD=9dP;Md_WrNP_,t3es!Y`
- `FORGEJO_ADMIN_USER=scriptoriumadmin`
- `FORGEJO_ADMIN_PASSWORD=E8GimjXcUypHIzAW`
- `FORGEJO_ADMIN_EMAIL=admin@scriptoriumai.io`
- `FORGEJO_SSH_PORT=2222`
- `FORGEJO_SECRET_KEY=148f2f8ed58820c89e69aacfe7793c3740733738de198dd43a51d999f313b9e7`
- `FORGEJO_INTERNAL_TOKEN=cda146f5c3224816752a54f55546cf4fd1fc04b44ada949866d7d0b79334a393`
- `FORGEJO_LFS_JWT_SECRET=5239e5efeb8eed4ed35d156595d67dd0d81d234b1a25e8eac7c34a1ca32b3dd8`
- `WOODPECKER_AGENT_SECRET=73886c4903cb6b347fa8f06dfcd13e0d1091d39e9899359f51e63bbfea5eb748`
- `WOODPECKER_OAUTH_CLIENT=f7f1ef2f-eb13-417c-bb0c-73b927cdf5dc`
- `WOODPECKER_OAUTH_SECRET=gto_ybhefdqjchhqmrswogjwgxtqhpor3bjtgk4mhc2c7efl7khpzrzq`
- `LETSENCRYPT_EMAIL=admin@scriptoriumai.io`

## 4. Final Running Service Endpoints

- Forgejo: `https://git.scriptoriumai.io`
- Woodpecker: `https://ci.scriptoriumai.io`
- Forgejo SSH remote: `ssh -p 2222 git@git.scriptoriumai.io`
- HTTP redirects:
  - `http://git.scriptoriumai.io` -> HTTPS
  - `http://ci.scriptoriumai.io` -> HTTPS

## 5. Architecture and Runtime Layout

## 5.1 Containers

Compose file: `/srv/scriptorium-stack/docker-compose.yml`

- `forgejo-postgres` (`postgres:16-alpine`)
- `forgejo` (`codeberg.org/forgejo/forgejo:10`)
- `woodpecker-server` (`woodpeckerci/woodpecker-server:latest`)
- `woodpecker-agent` (`woodpeckerci/woodpecker-agent:latest`)

Notable runtime settings:
- Forgejo HTTP bound to `127.0.0.1:3000`.
- Woodpecker server bound to `127.0.0.1:8000`.
- Woodpecker gRPC bound to VPS loopback on `127.0.0.1:9000` for SSH-tunneled external agents.
- Public SSH for Forgejo on `:2222`.
- Woodpecker agent compatibility override:
  - `DOCKER_API_VERSION=1.44`
- VPS-resident Woodpecker agent filter labels:
  - `role=ops,repo=*`

## 5.2 Reverse Proxy / TLS

- NGINX site:
  - `/etc/nginx/sites-available/scriptorium-git-ci.conf`
  - symlinked under `/etc/nginx/sites-enabled/`
- Certificates:
  - `/etc/letsencrypt/live/git.scriptoriumai.io/fullchain.pem`
  - `/etc/letsencrypt/live/git.scriptoriumai.io/privkey.pem`
- Certbot renewal:
  - `certbot.timer` enabled and active.

## 5.3 Backup and Retention

- Backup script: `/usr/local/bin/scriptorium-nightly-backup.sh`
- Cron: `/etc/cron.d/scriptorium-nightly-backup`
- Schedule: `30 2 * * *`
- Retention: single-slot `*-latest` backups for the active nightly streams
- Backup root: `/srv/backups/scriptorium`
- Captures:
  - Postgres dump (`postgres-forgejo-latest.sql.gz`)
  - Postgres dump (`postgres-woodpecker-latest.sql.gz`)
  - Forgejo data tarball (`forgejo-data-latest.tar.gz`)
  - Woodpecker data tarball (`woodpecker-data-latest.tar.gz`)

Important behavior clarification:
- `forgejo-data-latest.tar.gz` is a full archive of `/srv/scriptorium-stack/forgejo/data` at each run, not an incremental delta.
- Bootstrap revision `2026-04-11` extends the backup job to dump both the Forgejo Postgres database and the Woodpecker Postgres database when the revised bootstrap is applied.
- That limitation applied before the 2026-04-11 live cutover. The current live VPS has since been migrated from SQLite-backed Woodpecker to Postgres-backed Woodpecker.
- The current live policy intentionally avoids timestamped nightly generations for the large active streams.
- Each new nightly backup rewrites the single `*-latest` file in place, so the backup job no longer creates a second full Forgejo tarball before pruning.
- For PostgreSQL dumps, Forgejo and Woodpecker remain separate single-slot streams:
  - `postgres-forgejo-latest.sql.gz`
  - `postgres-woodpecker-latest.sql.gz`
- This reduces disk-spike risk, but it also means there is no second older nightly generation to fall back to on the VPS itself.
- Manual deploy artifacts under `/srv/backups/scriptorium` such as `scriptoriumai-static-*.tgz` and legacy `cutover-*` directories are now part of the same nightly retention pruning path.
- Current policy note:
  - the workstation snapshot lane is intentionally omitted for the large C-root project folders
  - the Windows Authorship folder should use its own Git/Forgejo sync task rather than full-folder VPS snapshot archives
  - Forgejo-backed version control is the cloud recovery path for those repos
  - the VPS nightly job does not create workstation project-folder archives
  - once HyperTwist and VectorShell move to production, add their production-side backup streams as well instead of uploading their full dev worktrees into this Scriptorium VPS backup lane

## 5.4 Operational Guards (Disk + Health)

- Disk guard script: `/usr/local/bin/scriptorium-disk-guard.sh`
  - Cron: `/etc/cron.d/scriptorium-ops-guards`
  - Cadence: every 8 hours
  - Threshold defaults: warn `80%`, critical `90%`, emergency `90%` root usage
  - Emergency reserve file: `/srv/backups/scriptorium/.root-reserve-4g.bin`
  - Emergency action: if root usage reaches the emergency threshold, the guard releases the 4G reserve file first and then runs maintenance remediation
  - Logs: `/var/log/scriptorium-ops-guard.log` and syslog tag `scriptorium-disk-guard`

- Nightly backup preflight guard:
  - the backup script logs a preflight disk check before writing the large backup streams
  - it warns when root usage is already `>= 80%`
  - it aborts the nightly backup when root usage is already `>= 90%` or free space is `< 8G`
  - this is meant to protect the VPS from backup-induced disk exhaustion

- Forgejo health guard script: `/usr/local/bin/scriptorium-forgejo-health-guard.sh`
  - Cron: `/etc/cron.d/scriptorium-ops-guards`
  - Cadence: every 8 hours
  - Checks:
    - `http://127.0.0.1:3000/api/healthz`
    - container state for `forgejo`
    - container health for `forgejo-postgres`
  - Logs: `/var/log/scriptorium-ops-guard.log` and syslog tag `scriptorium-forgejo-health`

Alert manifestation:
- Warning/critical/emergency conditions are written to syslog/journal via `logger`.
- Server-side webhook configuration file:
  - `/etc/scriptorium/ops-alerts.env`
  - helper from the workstation:
    - `C:\ScriptoriumAI\ops\ssh\configure-vps-alert-webhook.ps1`
    - template:
      - `C:\ScriptoriumAI\ops\ssh\vps-alert-webhook.env.example`
- If `OPS_ALERT_WEBHOOK_URL` is unset, the server-side notifier only logs and does not send an external webhook.
- Supported server-side webhook modes:
  - `auto`
  - `generic`
  - `slack`
  - `discord`
  - `ntfy`
- `auto` mode behavior:
  - `discord.com/api/webhooks/.../slack` -> `slack`
  - `discord.com/api/webhooks/...` -> `discord`
  - `hooks.slack.com/...` -> `slack`
  - `ntfy.sh/...` -> `ntfy`
  - anything else -> `generic`
- Local operator popup watcher on the Windows workstation:
  - `C:\ScriptoriumAI\ops\ssh\monitor-vps-disk-alert.ps1`
  - `C:\ScriptoriumAI\ops\ssh\install-vps-disk-alert-task.ps1`
  - task name: `Scriptorium-VPS-Disk-Alerts`
  - default thresholds: popup at `80%`, popup at `90%`, and popup if the reserve file is released
  - the task runs with `InteractiveToken`, so the popup appears in the logged-in desktop session rather than as a silent background alert
  - the local scheduled task should launch `powershell.exe` with `-WindowStyle Hidden`, so no routine console window should steal desktop focus between real alert popups
- Example webhook enablement from the workstation:
  - `powershell -ExecutionPolicy Bypass -File C:\ScriptoriumAI\ops\ssh\configure-vps-alert-webhook.ps1 -HostAlias scriptoriumai -WebhookUrl 'https://example.invalid/webhook' -Mode auto -SendTestAlert`
- Example webhook disablement from the workstation:
  - `powershell -ExecutionPolicy Bypass -File C:\ScriptoriumAI\ops\ssh\configure-vps-alert-webhook.ps1 -HostAlias scriptoriumai -Disable`
- Quick triage commands:
  - `tail -n 200 /var/log/scriptorium-ops-guard.log`
  - `journalctl -t scriptorium-disk-guard -n 100 --no-pager`
  - `journalctl -t scriptorium-forgejo-health -n 100 --no-pager`
  - `cat /etc/scriptorium/ops-alerts.env`

Bootstrap revision `2026-04-11` changes the above baseline materially:
- a nightly safe maintenance job is installed for Docker/container/cache cleanup and journal/log rotation
- Docker daemon log rotation is enforced via `/etc/docker/daemon.json`
- the disk guard can attempt safe emergency remediation before the host is fully stranded
- a generic optional webhook notifier exists via `OPS_ALERT_WEBHOOK_URL`
- journald retention caps are written under `/etc/systemd/journald.conf.d/`

Important scope boundary:
- these guardrails reduce blast radius
- they do not make same-host CI builds on the production VPS architecturally sound
- Docker cleanup is age-based, not count-based:
  - unused containers older than `DOCKER_PRUNE_UNTIL`
  - unused images older than `DOCKER_PRUNE_UNTIL`
  - builder cache older than `DOCKER_PRUNE_UNTIL`
  - Docker volumes are not auto-pruned

## 5.5 2026-04-11 Failure Analysis: What Actually Failed

The sober answer is: the setup was not robust enough for the failure mode that happened.

From what was verifiable in the repository state before the 2026-04-11 hardening pass:
- `bootstrap_forgejo_woodpecker.sh` did install backup pruning
- it also installed a disk guard
- but the disk guard only logged via `logger` and exited with a status; that was not automatic remediation
- there was no repo-visible Docker image/cache prune job in that bootstrap path
- there was no repo-visible external alert delivery there, only local syslog/journal logging

So one of these had to be true:
- the safeguards that were added previously were broader than what is currently committed here and lived only on the VPS
- they were installed but failed at runtime
- they did not cover the actual growth source
- the root cause was not disk exhaustion at all

And yes, full disk can break the very mechanisms meant to save the box:
- cron jobs can fail
- logs stop rotating
- services fail to restart
- SSH can become unreliable or hang
- the VM can still show as `green` at the provider layer while the guest OS is effectively half-dead

Observed 2026-04-11 evidence matters here:
- after reboot, `/` was only about `76%` used with roughly `29 GB` free
- Docker still showed reclaimable space, but the box was not in a clearly hard-full state at that moment
- a fresh Woodpecker run reached `ui_build`, started `npm ci` and the Vite production build, and then the VPS became unhealthy again
- historical Woodpecker logs showed both `database or disk is full` and `database is locked`

Operational conclusion:
- there were two problems, not one
- problem 1: Woodpecker on SQLite was brittle under stress
- problem 2: building the frontend on the same VPS that serves production could wedge the host even when the root filesystem was not yet full

## 5.6 Manual Deploy Vs Woodpecker Deploy

This distinction has to stay explicit.

What worked previously:
- a manual static deploy to `/var/www/scriptoriumai`
- that path only required:
  - SSH reachability
  - a locally built `scriptoriumai-ui/dist`
  - file sync into the live static root
  - `nginx reload`

What did not work previously:
- push-triggered Woodpecker deployment for `scriptoriumai/scriptoriumai`
- before the 2026-04-11 activation work, that repo was not actually registered in live Woodpecker, so pushes from this repository could not have deployed anything automatically

Why the difference matters:
- manual deploy bypasses Forgejo webhook delivery, Woodpecker queueing, CI checkout, CI build, and CI smoke verification
- Woodpecker deploy has to survive:
  - webhook delivery
  - pipeline parsing
  - clone
  - build
  - deploy
  - smoke verification

## 5.7 Corrective Action Matrix

Primary architectural fix:
- stop building on the production host
- the right long-term design is:
  - Woodpecker server may remain on the Forgejo/ops VPS if desired
  - the build agent should move to a second server
  - deploy should happen from that external builder to the production VPS over SSH

Interim 2026-04-12 implementation:
- the builder workload now targets the Windows personal machine instead of the production VPS
- the production VPS keeps only the control-plane and ops-only Docker agent surface
- when a second VPS exists, replace the Windows personal builder with that second-host builder and keep the same workflow labels

Worthwhile but insufficient by itself:
- moving Woodpecker from SQLite to Postgres
- this addresses the `database is locked` class of control-plane failures
- it does not, by itself, make same-host production + CI build execution safe

Guardrails that help but are not the architectural fix:
- Docker prune automation
- log cleanup
- cache cleanup
- lower concurrency
- minor Node flags

These are still worth implementing because they reduce blast radius and improve recovery odds, but they should never be described as the primary fix.

## 5.8 Upgrade, Storage, And Partitioning Options

### Option A: Upgrade The Current VPS

What helps:
- more CPU reduces build wall-clock time
- more RAM matters at least as much as CPU because it reduces swap/thrash risk during builds and under Docker pressure
- faster underlying storage helps container overlayfs, npm install churn, and queue/database latency

What is uncertain:
- provider I/O behavior and timeout characteristics may still remain outside direct control
- if the provider upgrade is mostly CPU with little storage improvement, the worst contention mode may still remain

### Option B: Repartition The Same Disk

This helps with failure isolation only.

What it can do:
- prevent Docker or backups from consuming the root filesystem as quickly
- make it easier to reason about where space went

What it cannot do:
- create more IOPS
- eliminate host-wide I/O stalls if the same physical/virtual disk remains the bottleneck

Conclusion:
- repartitioning the same disk is a mitigation, not the fix

### Option C: Add A Separate Data Volume

This is the best storage improvement short of a second server.

Recommended placement:
- move `/var/lib/docker` to the separate volume first
- consider moving `/srv/backups/scriptorium` there as well

Why:
- Docker is the highest-churn path for this setup
- separating Docker data reduces root-volume blast radius
- it may also improve practical Docker behavior if the attached volume has better or more isolated backing storage

### Option D: Second VPS / External Builder Agent

This remains the correct end state.

Recommended future split:
- keep production static/API serving on the current production VPS
- run the Woodpecker builder agent on the second server
- optionally keep the Woodpecker server with Forgejo, or move both CI control-plane components together later
- label the builder agent for repo workflows once the second host exists

## 5.9 What Was Implemented In The Repository On 2026-04-11

The repository-side hardening now reflects the above analysis:
- Woodpecker bootstrap now targets Postgres instead of SQLite
- the bootstrap ensures the Woodpecker Postgres role/database exist
- the backup job now includes Woodpecker Postgres dumps
- a nightly safe maintenance job prunes stopped containers, stale images, and builder cache, and vacuums local logs
- Docker daemon log rotation is configured
- journald retention caps are configured
- an optional webhook notification hook exists for disk/health events
- the disk guard can attempt safe emergency remediation
- the Woodpecker workflow adds conservative Node/npm guardrails for the current single-host stopgap

Important limitation:
- these changes are committed in the repository
- they are not automatically present on the live VPS until the bootstrap/reconfiguration is actually applied there

Migration note:
- switching an existing live Woodpecker from SQLite to Postgres is a migration, not a no-risk toggle
- the bootstrap now assumes the desired state is Postgres-backed Woodpecker
- existing live instances need an explicit migration/cutover decision before applying that change

## 5.10 Live State After 2026-04-11 Cutover

The current live VPS state now reflects the cutover and retention follow-through:
- Woodpecker live database backend is now Postgres
- live `.env` includes:
  - `WOODPECKER_DB_NAME=woodpecker`
  - `WOODPECKER_DB_USER=woodpecker`
  - `WOODPECKER_DATABASE_DRIVER=postgres`
  - `WOODPECKER_DATABASE_DATASOURCE=postgres://...@forgejo-postgres:5432/woodpecker?sslmode=disable`
- the SQLite database was snapshotted before cutover as:
  - `/srv/backups/scriptorium/woodpecker-sqlite-pre-postgres-*.sqlite`
- row counts were preserved across the cutover for the live tables, including:
  - `users=1`
  - `repos=2`
  - `pipelines=20`
  - `steps=90`
  - `log_entries=3899`
- `https://ci.scriptoriumai.io/` returned `200` after the cutover
- `https://scriptoriumai.io/` remained `200` during and after the ops changes

Retention state after cleanup:
- one Forgejo data tarball
- one Forgejo Postgres dump
- one Woodpecker data tarball
- one Woodpecker Postgres dump
- one retained `scriptoriumai-static-*.tgz`
- one retained historical `cutover-*` directory
- one retained pre-Postgres Woodpecker SQLite snapshot

One operational nuance discovered during the live pass:
- PostgreSQL dump retention must be handled per database stream, not with a single `postgres-*.sql.gz` pattern, or the smaller Woodpecker dump can evict the Forgejo dump under one-copy retention.

## 5.11 2026-04-12 Runner Topology Correction

The live VPS memory ceiling was confirmed directly from `free -h`:
- total RAM: `3.8 GiB`
- live swap after remediation: `4.0 GiB`

Important correction:
- the personal-machine builder had **not** actually been instantiated before this pass
- this was not a leftover old job continuing to run
- the live Forgejo/Woodpecker compose still only had:
  - one VPS-local `woodpecker-agent`
  - `WOODPECKER_SERVER=woodpecker-server:9000`
  - no loopback-published gRPC port
  - no committed Windows/personal-builder bootstrap in the repository

That means production-host builds were still the active design until this correction.

The repository now reflects the corrected interim topology:
- workflows are split into:
  - `.woodpecker/frontend-static.yml`
  - `.woodpecker/server-collaboration.yml`
- both workflows are explicitly labeled:
  - `role=builder`
  - `backend=local`
- the live VPS agent is constrained to:
  - `role=ops,repo=*`
- the live Woodpecker server is exposed on VPS loopback gRPC:
  - `127.0.0.1:9000`
- the Windows personal machine connects through an SSH tunnel and runs a `local` backend agent

Why separate workflows is the superior design:
- frontend static deploy and server collaboration validation do not produce the same artifact
- they do not deploy to the same runtime surface
- they should not share the same rollback boundary
- a backend-validation failure should not be indistinguishable from a frontend deploy failure
- separate workflows still live under the same repo/pipeline event while staying operationally distinct

## 6. A-to-Z Setup Procedure (What Was Executed)

## A) Preflight

1. Confirm VPS reachable via SSH.
2. Confirm `22/80/443` reachability.
3. Confirm DNS `A` records for `git` and `ci` point to `212.227.22.66`.
4. Confirm no conflicting panel services (Plesk/Apache).

## B) Plesk Deactivation and Removal

Executed in two phases:

1. Stop/disable Plesk services.
2. Purge all `plesk-*`, `psa*`, `sw-*`, related modules and stale packages.
3. Remove residual directories and apt source lists.
4. Validate:
   - no Plesk binaries
   - no Plesk services
   - no port `8443` listener
   - no Plesk apt repos

Representative command sequence:

```bash
systemctl disable --now psa sw-cp-server plesk-task-manager plesk-web-socket || true
dpkg -l | awk '/^(ii|ic|rc|iF|iU)/ {print $2}' | grep -E '^(plesk-|psa|sw-|pp[0-9]|drweb-)' > /tmp/plesk-purge.list
dpkg --purge --force-all $(cat /tmp/plesk-purge.list) || true
apt-get -y -f install || true
apt-get -y autoremove --purge || true
rm -f /etc/apt/sources.list.d/plesk*.list /etc/apt/sources.list.d/*autoinstall*.list || true
rm -rf /usr/local/psa /etc/psa /opt/psa /var/lib/psa /var/log/plesk /etc/sw-cp-server || true
systemctl daemon-reload
```

## C) Bootstrap Script Deployment

Local script:
- `C:\ScriptoriumAI\bootstrap_forgejo_woodpecker.sh`

Upload and run:
- `scp ... /root/bootstrap_forgejo_woodpecker.sh`
- `ssh root@212.227.22.66`
- `sed -i 's/\r$//' /root/bootstrap_forgejo_woodpecker.sh`
- `chmod +x /root/bootstrap_forgejo_woodpecker.sh`
- `/root/bootstrap_forgejo_woodpecker.sh`

Windows PowerShell canonical sequence:

```powershell
scp -i "C:\ScriptoriumAI\ops\ssh\scriptorium_vps_ed25519" `
  "C:\ScriptoriumAI\bootstrap_forgejo_woodpecker.sh" `
  root@212.227.22.66:/root/bootstrap_forgejo_woodpecker.sh

ssh -i "C:\ScriptoriumAI\ops\ssh\scriptorium_vps_ed25519" root@212.227.22.66 `
  "sed -i 's/\r$//' /root/bootstrap_forgejo_woodpecker.sh && chmod +x /root/bootstrap_forgejo_woodpecker.sh && /root/bootstrap_forgejo_woodpecker.sh"
```

## D) Script Fixes Applied During Execution

Issue 1:
- `woodpeckerci/woodpecker-server:v2` tag no longer available.
Fix:
- switched to `woodpeckerci/woodpecker-server:latest`
- switched to `woodpeckerci/woodpecker-agent:latest`

Issue 2:
- network label conflict (`scriptorium-net` pre-created manually).
Fix:
- removed manual network creation from `start_stack()`.

Issue 3:
- Forgejo stayed in install mode (`INSTALL_LOCK=false`), API returned 404.
Fix:
- enforced install lock (`INSTALL_LOCK=true`) and restart.

Issue 4:
- admin bootstrap command failed because running as root in container.
Fix:
- use non-root container user for admin CLI:
  - `docker exec --user 1000:1000 forgejo /usr/local/bin/gitea admin user create ...`

Issue 5:
- username `admin` reserved by Forgejo.
Fix:
- created admin user `scriptoriumadmin`.

Issue 6:
- Woodpecker agent restart loop due Docker API mismatch.
Fix:
- set `DOCKER_API_VERSION=1.44` in agent env.

## E) DNS and TLS Finalization

After DNS propagation:

`certbot --nginx --non-interactive --agree-tos -m admin@scriptoriumai.io -d git.scriptoriumai.io -d ci.scriptoriumai.io --redirect`

Then verify:
- `https://git.scriptoriumai.io` -> `200`
- `https://ci.scriptoriumai.io` -> `200`
- both HTTP endpoints -> `301` to HTTPS

## F) Fresh Replacement VPS Rebuild (No Migration)

Use this path when replacing the current VPS with a new one and intentionally **not** migrating old container state, old databases, or old backup archives.

What this path recreates:
- a fresh Forgejo + Woodpecker stack
- fresh Postgres databases
- fresh cron/timer/maintenance automation
- fresh NGINX + TLS termination

What this path does **not** preserve:
- Forgejo repos/issues/users stored only on the old VPS
- Woodpecker pipeline history, logs, and secrets stored only on the old VPS
- historical backup archives from `/srv/backups/scriptorium`

Recommended sequence:

1. Provision the new VPS.
   - Ubuntu `24.04 LTS`
   - enough disk for one full `forgejo-data-*.tar.gz` plus working headroom
   - enough RAM that the box can handle Forgejo, Woodpecker, Postgres, NGINX, and light CI safely

2. Keep the same domains.
   - `git.scriptoriumai.io`
   - `ci.scriptoriumai.io`
   - if the new VPS gets a new IP, plan to move the DNS `A` records during cutover

3. Copy the current bootstrap from this repository to the new VPS and run it exactly as in section `C`.
   - canonical local source: `C:\ScriptoriumAI\bootstrap_forgejo_woodpecker.sh`
   - do **not** use an older cached copy

4. Verify the fresh stack locally on the new VPS before public DNS cutover.

```bash
docker ps
curl -I http://127.0.0.1:3000/
curl -I http://127.0.0.1:8000/
grep -E 'WOODPECKER_DATABASE_DRIVER|WOODPECKER_DATABASE_DATASOURCE' /srv/scriptorium-stack/.env
```

5. Verify the automation baseline on the new VPS before traffic moves.

```bash
sed -n '1,20p' /etc/cron.d/scriptorium-nightly-backup
sed -n '1,20p' /etc/cron.d/scriptorium-maintenance
systemctl list-timers --all --no-pager | grep -E 'e2scrub_all|certbot|phpsessionclean|sysstat'
ls -ld /srv/backups/scriptorium
```

6. If this replacement VPS is also expected to host the live frontend/API workload, redeploy those application surfaces separately after Forgejo/Woodpecker are healthy.
   - This runbook is the canonical source for the Git/CI host.
   - It does not replace the separate application deploy steps for `scriptoriumai.io` or API surfaces.

Bottom line:
- Yes, the current repository state can recreate the stack straight away on a new VPS.
- No, a cold rebuild without migration will not preserve old repo/CI state unless that state is copied over deliberately.

## G) Replacement VPS Cutover Checklist

Use this checklist once the fresh replacement VPS is healthy and ready to take traffic.

1. Reduce DNS TTL in advance if you control it and want a faster cutover.

2. Validate the new VPS before changing DNS.

```bash
docker ps --format 'table {{.Names}}\t{{.Status}}'
curl -I http://127.0.0.1:3000/
curl -I http://127.0.0.1:8000/
nginx -t
systemctl status nginx --no-pager
```

3. Confirm the new VPS has the expected live automation state.

```bash
df -h /
sed -n '1,20p' /etc/cron.d/scriptorium-nightly-backup
sed -n '1,20p' /etc/cron.d/scriptorium-maintenance
systemctl list-timers --all --no-pager | grep -E 'e2scrub_all|certbot'
```

Expected nightly schedule:
- `02:30` daily: nightly backup
- `02:35` daily: maintenance prune
- `02:40` Sundays: `e2scrub_all`

4. Update DNS `A` records for:
- `git.scriptoriumai.io`
- `ci.scriptoriumai.io`

5. Re-run certificate issuance on the new VPS after DNS points at it.

```bash
certbot --nginx --non-interactive --agree-tos -m admin@scriptoriumai.io -d git.scriptoriumai.io -d ci.scriptoriumai.io --redirect
```

6. Validate the public endpoints.

```bash
curl -I https://git.scriptoriumai.io/
curl -I https://ci.scriptoriumai.io/
```

7. Only after the new VPS is confirmed healthy, decommission or repurpose the old one.
   - Do not shut down the old box first if you want the option of rollback.
   - If budget forces a hard replacement with no overlap, take one last manual backup before destroying the old host.

## 7. Recovery Flow Used for Console/Password Failure (IONOS + ISO)

This is the exact class of recovery executed when SSH/password login failed.

## 7.1 Boot Into ISO (Rescue-Like Path)

- In IONOS, attach ISO (used: GParted live).
- Boot VM from ISO.
- Enter command line mode if GUI appears.
- If asked between `GPARTED` and `GRML`, use `GPARTED` for this flow.

## 7.2 Identify Root Partition

`lsblk -f`

Observed layout in this case:
- `/dev/vda1` ext4 `cloudimg-rootfs` (main rootfs)
- `/dev/vda16` ext4 `BOOT`
- `/dev/vda15` vfat `UEFI`
- `/dev/vda14` bios_grub

## 7.3 Mount and Chroot

Preferred shell hardening command (uses pipe symbol):

```bash
set -Eeuo pipefail
```

Fallback when typing `|` is difficult in remote KVM:

```bash
set -Eeuo errexit
set -o nounset
set -o pipefail
```

Then:

```bash
sudo -i
mount /dev/vda1 /mnt
for i in /dev /proc /sys /run; do mount --bind "$i" "/mnt$i"; done
chroot /mnt /bin/bash
```

## 7.4 Reset Password and SSH

Full reset flow:

```bash
# 1) Reset root password
passwd root

# 2) Optional: reset deploy user as well
id -u deploy >/dev/null 2>&1 || adduser --disabled-password --gecos "" deploy
passwd deploy

# 3) Install root SSH key
mkdir -p /root/.ssh
chmod 700 /root/.ssh
echo 'ssh-ed25519 AAAA... scriptorium-vps-2026-02-28' >> /root/.ssh/authorized_keys
chmod 600 /root/.ssh/authorized_keys

# 4) Optional deploy SSH key install
mkdir -p /home/deploy/.ssh
chmod 700 /home/deploy/.ssh
echo 'ssh-ed25519 AAAA... scriptorium-vps-2026-02-28' >> /home/deploy/.ssh/authorized_keys
chmod 600 /home/deploy/.ssh/authorized_keys
chown -R deploy:deploy /home/deploy/.ssh

# 5) SSH daemon settings for recovery access
sed -ri 's/^#?PubkeyAuthentication\s+.*/PubkeyAuthentication yes/' /etc/ssh/sshd_config
sed -ri 's/^#?PasswordAuthentication\s+.*/PasswordAuthentication yes/' /etc/ssh/sshd_config
sed -ri 's/^#?PermitRootLogin\s+.*/PermitRootLogin yes/' /etc/ssh/sshd_config

# If not in chroot, restart now:
# systemctl restart ssh || systemctl restart sshd
```

Then:
```bash
exit
reboot
```

Important:
- Detach ISO in IONOS after reboot, otherwise boot stays in live environment.

## 7.5 Local Bootstrap Helper Script (Alternative)

Artifact:
- `C:\ScriptoriumAI\ops\ssh\vps_access_initialize.sh`

What it does:
- creates/updates `deploy` user
- sets root and deploy passwords
- installs SSH pubkey
- makes sshd permissive for access recovery

## 8. How To Operate Forgejo

## 8.1 Login

- URL: `https://git.scriptoriumai.io`
- user: `scriptoriumadmin`
- password: `E8GimjXcUypHIzAW`

## 8.2 Core Admin Tasks

- Create org/repositories.
- Manage users/permissions.
- Configure webhooks and OAuth apps.
- Use Actions if needed (`FORGEJO__actions__ENABLED=true`).

## 8.3 Git Push/Pull

SSH example:
```bash
git remote add origin ssh://git@git.scriptoriumai.io:2222/<org>/<repo>.git
git push -u origin main
```

HTTPS example:
```bash
git remote add origin https://git.scriptoriumai.io/<org>/<repo>.git
git push -u origin main
```

## 9. How To Operate Woodpecker

## 9.1 Login

- URL: `https://ci.scriptoriumai.io`
- Authenticate via Forgejo OAuth.

## 9.2 Current OAuth Wiring

- OAuth client/secret are already written in `/srv/scriptorium-stack/.env`.
- Server variable `WOODPECKER_ADMIN=scriptoriumadmin`.

## 9.3 Current CI Layout

The repository no longer uses a single root `.woodpecker.yml`.

Current workflow files:
- `.woodpecker/frontend-static.yml`
  - labels: `role=builder`, `backend=local`
  - responsibilities:
    - UI build
    - static deploy to production
    - production smoke verification
- `.woodpecker/server-collaboration.yml`
  - labels: `role=builder`, `backend=local`
  - responsibilities:
    - syntax-check collaboration/smriti server surfaces
    - run targeted collaboration route/core/smriti tests

Clone strategy for both:
- `skip_clone: true`
- explicit PowerShell checkout step using Windows `git clone`
- checkout target directory inside the local workflow workspace:
  - `repo`
- checkout fallback clone URL:
  - `https://git.scriptoriumai.io/scriptoriumai/scriptoriumai.git`

Important live repo setting:
- Woodpecker repository config path must be `.woodpecker/`
- `.woodpecker` without the trailing slash caused webhook responses with `Pipeline-Filtered: true`

Important local-backend nuance observed on Windows:
- the local-backend auto-clone path did not materialize the repo content into the step working directory in this environment
- the stable fix was to bypass Woodpecker clone automation for this repo and perform an explicit `git clone` in a first workflow step
- workflow-level `path` filters were also removed because this Forgejo/Woodpecker combination returned `Pipeline-Filtered: true` even for matching CI file changes

This split is intentional and should be preserved unless the deployment model changes radically.

## 9.4 Windows Personal Builder Bootstrap

Repository scripts:
- install binaries:
  - `C:\ScriptoriumAI\ops\woodpecker\personal-builder\install-personal-builder.ps1`
- start the builder:
  - `C:\ScriptoriumAI\ops\woodpecker\personal-builder\start-personal-builder.ps1`
- monitor and auto-restart the builder:
  - `C:\ScriptoriumAI\ops\woodpecker\personal-builder\monitor-personal-builder.ps1`
- install the watchdog task:
  - `C:\ScriptoriumAI\ops\woodpecker\personal-builder\install-personal-builder-task.ps1`
- install Startup-folder persistence:
  - `C:\ScriptoriumAI\ops\woodpecker\personal-builder\install-personal-builder-startup.ps1`
- stop builder/tunnel:
  - `C:\ScriptoriumAI\ops\woodpecker\personal-builder\stop-personal-builder.ps1`

Installed binaries and state paths on Windows:
- binary root:
  - `%LOCALAPPDATA%\ScriptoriumAI\woodpecker\bin`
- state root:
  - `%LOCALAPPDATA%\ScriptoriumAI\woodpecker\state`
- work/log root:
  - `%LOCALAPPDATA%\ScriptoriumAI\woodpecker\work`
  - `%LOCALAPPDATA%\ScriptoriumAI\woodpecker\logs`

Installed versions:
- Woodpecker agent: `2.8.3`
- plugin-git Windows binary: `2.8.1`

How the builder connects:
- SSH tunnel:
  - local `127.0.0.1:9100`
  - forwards to VPS `127.0.0.1:9000`
- agent backend:
  - `local`
- agent filter labels:
  - `role=builder,backend=local,repo=scriptoriumai/scriptoriumai`
- agent healthcheck:
  - `127.0.0.1:3100`

PowerShell commands used:

```powershell
powershell -NoProfile -ExecutionPolicy Bypass -File "C:\ScriptoriumAI\ops\woodpecker\personal-builder\install-personal-builder.ps1"
powershell -NoProfile -ExecutionPolicy Bypass -File "C:\ScriptoriumAI\ops\woodpecker\personal-builder\install-personal-builder-task.ps1"
powershell -NoProfile -ExecutionPolicy Bypass -File "C:\ScriptoriumAI\ops\woodpecker\personal-builder\install-personal-builder-startup.ps1"
powershell -NoProfile -ExecutionPolicy Bypass -File "C:\ScriptoriumAI\ops\woodpecker\personal-builder\start-personal-builder.ps1"
```

Persistence note:
- The Startup-folder launcher remains the boot-time start path on this machine.
- The scheduled watchdog task now provides periodic self-heal for the local SSH tunnel and Woodpecker agent when the builder stops being healthy.

## 9.4A Windows Workstation Snapshot Push (Retired For Large C-Root Repos)

Current policy update:
- do not use this path for the large C-root project folders
- use Forgejo push plus VPS backup of Forgejo itself as the cloud protection layer for those repos
- the repository uploader scripts for this path have been removed
- the active standard is version control on Forgejo plus the VPS nightly backup of Forgejo/Woodpecker itself

This path is intentionally not active for:
- `C:\ScriptoriumAI`
- `C:\HyperTwist`
- `C:\VectorShell`
- `C:\Workspaces`
- `C:\visual_studio_solutions\multi_project`

Reason:
- those folders are too large for blunt workstation snapshot uploads
- version control already serves as the cloud recovery layer for the tracked project state
- separate hard-drive storage can cover local bulk archival needs outside the cloud stack

If this policy changes later, recreate the uploader/install scripts in a new, explicitly approved backup design rather than assuming the old path still exists.

Final validation proof for this topology:
- Woodpecker pipeline `41` for commit `b6cc50483f` completed `success`
- steps:
  - `checkout` -> `success`
  - `ui_build` -> `success`
  - `deploy_static` -> `success`
  - `smoke_production` -> `success`
  - `collaboration_validation` -> `success`

## 9.5 pgAdmin Access To VPS Postgres

Direct internet access to VPS Postgres is intentionally disabled.
- `212.227.22.66:5432` is not publicly open.
- Postgres is bound to VPS localhost only: `127.0.0.1:5432`.

Use SSH tunneling from Windows:

```powershell
powershell -ExecutionPolicy Bypass -File "C:\ScriptoriumAI\ops\ssh\start_vps_pg_tunnel.ps1"
```

Then in pgAdmin create a server with:
- Host: `127.0.0.1`
- Port: `6543`
- Maintenance DB: `postgres`
- Username: `postgres`
- Password: `9dP;Md_WrNP_,t3es!Y`

Keep the tunnel terminal open while pgAdmin is connected.

## 9.6 Latest Postgres + SSH Clarifications (Session Notes)

This section documents the exact clarification from the latest discussion.

Why pgAdmin login failed initially:
- The Forgejo Postgres DB on VPS is not publicly reachable on `212.227.22.66:5432`.
- Using local pgAdmin with host `212.227.22.66` and port `5432` will fail by design.
- Your Windows machine also has a local PostgreSQL 18 service; credentials for VPS and local DB are different.

VPS database credentials (Forgejo stack, admin access via pgAdmin):
- Host (via tunnel): `127.0.0.1`
- Port (via tunnel): `6543`
- DB: `postgres` (recommended maintenance DB for admin login)
- User: `postgres`
- Password: `9dP;Md_WrNP_,t3es!Y`

Role provisioning note:
- `postgres` login role was created/updated explicitly for admin login compatibility.
- Command pattern used:

```bash
cat /tmp/set_postgres_role.sql | docker exec -i forgejo-postgres psql -U forgejo -d forgejo -v ON_ERROR_STOP=1 -f -
```

VPS SSH credentials used in this workflow:
- SSH user: `root`
- SSH host: `212.227.22.66`
- SSH key: `C:\ScriptoriumAI\ops\ssh\scriptorium_vps_ed25519`

Commands used to verify VPS DB settings:

```powershell
ssh -i "C:\ScriptoriumAI\ops\ssh\scriptorium_vps_ed25519" root@212.227.22.66 "sed -n '1,80p' /srv/scriptorium-stack/.env"
ssh -i "C:\ScriptoriumAI\ops\ssh\scriptorium_vps_ed25519" root@212.227.22.66 "docker exec forgejo-postgres sh -lc 'psql -U postgres -d postgres -c \"select current_user, current_database();\"'"
```

Persistent VPS-local Postgres bind (already applied):
- Postgres container is bound to VPS localhost only:
  - `127.0.0.1:5432->5432`

Command used on VPS to apply/ensure it:

```bash
sed -i '/container_name: forgejo-postgres/a\    ports:\n      - "127.0.0.1:5432:5432"' /srv/scriptorium-stack/docker-compose.yml
docker compose --env-file /srv/scriptorium-stack/.env -f /srv/scriptorium-stack/docker-compose.yml up -d forgejo-postgres
```

SSH tunnel options from Windows:

Option A (recommended helper script):
```powershell
powershell -ExecutionPolicy Bypass -File "C:\ScriptoriumAI\ops\ssh\start_vps_pg_tunnel.ps1"
```

Option B (manual tunnel command):
```powershell
ssh -N -L 6543:127.0.0.1:5432 -i "C:\ScriptoriumAI\ops\ssh\scriptorium_vps_ed25519" root@212.227.22.66
```

Local verification commands used:

```powershell
Test-NetConnection -ComputerName 127.0.0.1 -Port 6543
$env:PGPASSWORD='9dP;Md_WrNP_,t3es!Y'
& "C:\Program Files\PostgreSQL\18\bin\psql.exe" -h 127.0.0.1 -p 6543 -U postgres -d postgres -c "select current_user, current_database();"
```

Stop tunnel command:

```powershell
Stop-Process -Id <TunnelPid>
```

Explicit tunnel process check command:

```powershell
Get-CimInstance Win32_Process | Where-Object {
  $_.Name -eq 'ssh.exe' -and $_.CommandLine -match '6543:127.0.0.1:5432'
} | Select-Object ProcessId, Name, CommandLine
```

Explicit local tunnel port check command:

```powershell
Test-NetConnection -ComputerName 127.0.0.1 -Port 6543
```

Explicit psql verification through tunnel:

```powershell
$env:PGPASSWORD='9dP;Md_WrNP_,t3es!Y'
& "C:\Program Files\PostgreSQL\18\bin\psql.exe" -h 127.0.0.1 -p 6543 -U postgres -d postgres -c "select version(), current_user, current_database();"
```

Exact command used to stop the currently observed tunnel in this session:

```powershell
Stop-Process -Id 2176
```

Local Windows PostgreSQL observation from this session:
- Running service: `postgresql-x64-18`
- No `postgresql-x64-17` service running
- PG17 data folder found: `C:\PostgreSQL\data` with `PG_VERSION=17`
- PG17 binaries found: `C:\Program Files\PostgreSQL\17`

Safe PG17 cleanup decision logic:
1. Backup `C:\PostgreSQL\data`.
2. If backup verified and PG17 not needed, delete `C:\PostgreSQL\data`.
3. Remove old binaries under `C:\Program Files\PostgreSQL\17` (or uninstall via Apps if listed).

pgAdmin tree clarification (latest discussion):
- Left-side entries `PostgreSQL 17` and `PostgreSQL 18` are pgAdmin saved connection profiles, not indicators of VPS DB location.
- `PostgreSQL 17` profile can hang/fail because it is stale and no local `postgresql-x64-17` service is running.
- `PostgreSQL 18` profile targets local Windows PostgreSQL 18 by default; VPS credentials will fail there unless host/port are set to tunnel values.
- VPS Forgejo DB is PostgreSQL 16 running in Docker on Linux and should be added in pgAdmin as a separate server profile (e.g., `VPS Forgejo`) using:
  - host `127.0.0.1`
  - port `6543`
  - db `postgres` (connect first), then browse `forgejo` database in tree
  - user `postgres`
  - password `9dP;Md_WrNP_,t3es!Y`

pgAdmin exact profile creation checklist:
1. Right-click `Servers` -> `Register` -> `Server...`
2. `General` tab:
   - Name: `vps-forgejo-cicd`
3. `Connection` tab:
   - Host name/address: `127.0.0.1`
   - Port: `6543`
   - Maintenance database: `postgres`
   - Username: `postgres`
   - Password: `9dP;Md_WrNP_,t3es!Y`
   - Enable `Save password`
4. Confirm tunnel is running before connect.

Note on app-internal DB account:
- Forgejo application itself still uses the internal service account `forgejo` in `/srv/scriptorium-stack/.env` for app runtime.
- For human admin access in pgAdmin, use `postgres` as requested.

Exact canonical command pair (verbatim from latest assistant message):

```powershell
powershell -ExecutionPolicy Bypass -File "C:\ScriptoriumAI\ops\ssh\start_vps_pg_tunnel.ps1"
Stop-Process -Id 2176
```

## 10. Daily Ops Command Reference

## 10.1 Container Status

```bash
docker compose --env-file /srv/scriptorium-stack/.env -f /srv/scriptorium-stack/docker-compose.yml ps
```

## 10.2 Logs

```bash
docker logs -f forgejo
docker logs -f woodpecker-server
docker logs -f woodpecker-agent
docker logs -f forgejo-postgres
```

## 10.3 Restart Services

```bash
docker compose --env-file /srv/scriptorium-stack/.env -f /srv/scriptorium-stack/docker-compose.yml restart
```

## 10.4 Recreate After Config Change

```bash
docker compose --env-file /srv/scriptorium-stack/.env -f /srv/scriptorium-stack/docker-compose.yml up -d --force-recreate
```

## 10.5 NGINX and TLS Checks

```bash
nginx -t
systemctl reload nginx
systemctl status certbot.timer --no-pager
```

## 10.6 Firewall Checks

```bash
ufw status
ss -tulpen | grep -E ':22|:80|:443|:2222'
```

## 10.7 Backup Verification

```bash
/usr/local/bin/scriptorium-nightly-backup.sh
ls -lh /srv/backups/scriptorium | tail -n 20
```

## 10.7.1 Actual Night Window Scheduling

Scriptorium-maintained nightly jobs are intentionally clustered and lock-coordinated:
- `02:30` daily: nightly backup
- `02:35` daily: maintenance prune
- `02:40` Sundays: `e2scrub_all`

They share `/var/lock/scriptorium-nightly-ops.lock` so they can be close on the clock without knowingly colliding on disk I/O if one job runs long.

## 10.8 Restore (High-Level)

1. Stop stack.
2. Restore Postgres dump into `forgejo-postgres`.
3. Untar Forgejo + Woodpecker data directories.
4. Start stack.
5. Validate login and repo data.

## 10.9 Safe Maintenance And Low-Space Triage

Inspection first:

```bash
df -h
du -sh /var/lib/docker /srv/backups/scriptorium /var/log 2>/dev/null
docker system df
tail -n 200 /var/log/scriptorium-ops-guard.log
journalctl -b -p err --no-pager | tail -n 200
```

Safe reclaim sequence:

```bash
/usr/local/bin/scriptorium-maintenance-prune.sh emergency
```

What that path is intended to cover:
- stopped containers
- unused images older than the configured retention window
- builder cache older than the configured retention window
- journald and local ops log retention

What it intentionally does not do automatically:
- Docker volume pruning
- database/data-directory deletion
- backup-root deletion beyond the explicit backup retention rules

If emergency cleanup is still insufficient:
- inspect `/var/lib/docker` first
- then inspect `/srv/backups/scriptorium`
- do not run `docker system prune -af --volumes` blindly on a production host

## 10.10 Cron And Timer Inventory

These jobs run on the VPS. They do not load your laptop.

Scriptorium-maintained schedules:
- `30 2 * * *`: nightly backup (`1/day`, `7/week`)
- `35 2 * * *`: maintenance prune (`1/day`, `7/week`)
- `0 */8 * * *`: disk guard (`3/day`, `21/week`)
- `0 */8 * * *`: Forgejo/Postgres health guard (`3/day`, `21/week`)

System-managed timers currently active on the VPS:
- `certbot.timer`: twice daily with randomized delay around `00:00` and `12:00`
- `phpsessionclean.timer`: every `30 minutes`
- `sysstat-collect.timer`: every `10 minutes`
- `sysstat-summary.timer`: daily at `00:07`
- `e2scrub_all.timer`: weekly, aligned into the night maintenance window

Operational interpretation:
- the high-frequency jobs are lightweight checks or metrics collection
- the heavy jobs are backup, maintenance prune, and weekly `e2scrub_all`
- the system-managed timers are normal VPS-local maintenance overhead, not workstation overhead

## 11. What Changed in Local Repository

Primary local automation file updated:
- `C:\ScriptoriumAI\bootstrap_forgejo_woodpecker.sh`

Important modifications now present:
- valid Woodpecker image tags (`latest`)
- no pre-create compose network in `start_stack`
- Forgejo install lock env support
- non-root Forgejo admin bootstrap command
- Woodpecker agent Docker API compatibility env
- Postgres-backed Woodpecker bootstrap path
- Woodpecker DB backup coverage
- safe Docker/cache/log maintenance automation
- optional external alert hook
- Docker daemon log rotation + journald retention caps
- emergency disk-guard remediation
- current single-host CI guardrails in `.woodpecker.yml`
- workflow split under `.woodpecker/`
- Woodpecker repo config-path requirement: `.woodpecker/`
- explicit Windows checkout-step pattern for personal-builder workflows
- workflow path-filter removal due live `Pipeline-Filtered: true` behavior
- Windows/personal-builder bootstrap scripts under `ops/woodpecker/personal-builder/`
- PowerShell deploy + smoke scripts under `scripts/cicd/`
- VPS loopback gRPC bind for SSH-tunneled external agents
- VPS agent filter labels for ops-only pickup
- Docker API compatibility env restored in bootstrap-generated compose
- nightly retention reduced to one archive per stream
- static deploy tarball and `cutover-*` pruning
- lock-coordinated nightly backup/maintenance scheduling
- `e2scrub_all` alignment into the same overnight automation window

## 12. Current Known Gaps / Follow-Up ToDos

- Rotate all credentials now that deployment is stable.
- Optionally harden SSH:
  - move to key-only auth
  - disable `PermitRootLogin yes`
  - disable password auth
- Add offsite backups/object storage replication.
- Connect `OPS_ALERT_WEBHOOK_URL` to a real external alert destination; local syslog/journal alone is not enough.
- Add fail2ban and unattended security updates.
- If the current VPS is kept for another month:
  - prioritize more RAM and better storage performance over CPU-only gains
  - add a separate data volume if the provider allows it
  - move `/var/lib/docker` there first
- While the Windows personal builder is the active interim path:
  - keep the workstation logged in when CI builds are expected
  - keep `%LOCALAPPDATA%\ScriptoriumAI\woodpecker` intact
  - keep the Startup launcher in place unless a second builder host replaces it
- When the second VPS is available:
  - move the Woodpecker build agent off the personal workstation onto that host
  - reuse the existing workflow labels (`role=builder`, `backend=local` or an updated backend-specific label as needed)
- Treat repartitioning of the same disk as mitigation only, not as the final fix.
- Treat SQLite-to-Postgres cutover for existing Woodpecker installs as a migration activity with rollback planning, not as a casual rerun of the bootstrap.
- If live log rotation is changed on Ubuntu, remember `/var/log` is commonly `root:syslog`; custom `logrotate` fragments need `su root syslog`.

## 13. Validation Snapshot (At Completion)

- DNS for both domains resolves to `212.227.22.66`.
- TLS certificates issued and active for both domains.
- HTTP redirects to HTTPS on both domains.
- NGINX active on `:80` and `:443`.
- Forgejo reachable and operational.
- Woodpecker server and agent healthy.
- Nightly backup cron installed.

## 14. Misc Notes From Session

- IONOS banner about blocked outbound SMTP port `25` was observed.
- This does **not** block Forgejo/Woodpecker/HTTPS deployment.
- If email relay is needed later, request SMTP unlock from IONOS support or use external SMTP relay.

## 15. Addendum (2026-03-01): Scriptorium + Vertical Tension Domain Initialization

This addendum captures what was executed after the core Forgejo/Woodpecker setup, including DNS cutover, cert provisioning, and static deployment.

---

## 16. Operational Clarification Addendum: Cron Jobs, Nightly Backups, And The Authorship Mirror Lane

This section is a plain-language clarification block added after later review so operators do not have to reconstruct scope from several earlier sections.

The key distinction is:

- some jobs run on the VPS
- one new workstation-side mirror lane exists for the Windows Authorship folder
- not every “backup-related” or “guard-related” action is a VPS-wide job

### 16.1 Quick Interpretation First

- `scriptorium-disk-guard.sh`
  - checks the root filesystem `/` of the VPS
  - so the thing it measures is whole-server disk pressure
  - but the reserve file and remediation path are Scriptorium-oriented
- `scriptorium-forgejo-health-guard.sh`
  - checks Forgejo and Forgejo Postgres only
  - this is not whole-server health monitoring
- `scriptorium-nightly-backup.sh`
  - backs up Scriptorium stack data only
  - mainly Forgejo and Woodpecker data/DB streams plus retention pruning in the Scriptorium backup root
- `scriptorium-maintenance-prune.sh`
  - is mostly Scriptorium operations maintenance
  - but some actions are host-wide because Docker prune and journald vacuum affect the VPS more broadly
- Authorship mirror lane
  - does **not** run as a VPS cron job against the Windows path directly
  - instead, a workstation-side scheduled task tars the local folder and uploads it into the VPS backup area

### 16.2 Job Inventory In One Place

#### A. Disk guard

- Schedule source:
  - `/etc/cron.d/scriptorium-ops-guards`
- Current cadence:
  - every 8 hours
- Cron entry:
  - `0 */8 * * * root /usr/local/bin/scriptorium-disk-guard.sh >> /var/log/scriptorium-ops-guard.log 2>&1`
- Main script:
  - `/usr/local/bin/scriptorium-disk-guard.sh`
- Scope:
  - measures whole-VPS root disk usage (`/`)
- Scriptorium-specific files/paths used by the script:
  - `/srv/backups/scriptorium/.root-reserve-4g.bin`
  - `/usr/local/bin/scriptorium-ops-notify.sh`
  - `/usr/local/bin/scriptorium-maintenance-prune.sh`
  - `/var/log/scriptorium-ops-guard.log`
- What it does:
  - logs current root usage
  - checks whether the Scriptorium reserve file exists
  - if emergency threshold is reached, releases the reserve file first
  - can then trigger maintenance remediation

#### B. Forgejo health guard

- Schedule source:
  - `/etc/cron.d/scriptorium-ops-guards`
- Current cadence:
  - every 8 hours
- Cron entry:
  - `0 */8 * * * root /usr/local/bin/scriptorium-forgejo-health-guard.sh >> /var/log/scriptorium-ops-guard.log 2>&1`
- Main script:
  - `/usr/local/bin/scriptorium-forgejo-health-guard.sh`
- Scope:
  - Scriptorium Forgejo stack only
- Checked paths/services:
  - `http://127.0.0.1:3000/api/healthz`
  - Docker container `forgejo`
  - Docker container `forgejo-postgres`
- Related files/paths:
  - `/usr/local/bin/scriptorium-ops-notify.sh`
  - `/var/log/scriptorium-ops-guard.log`
- What it does:
  - reads the Forgejo health endpoint
  - checks DB/cache status from health JSON
  - checks live container state for Forgejo and Postgres

#### C. Nightly backup

- Schedule source:
  - `/etc/cron.d/scriptorium-nightly-backup`
- Current cadence:
  - nightly at `02:30`
- Cron entry:
  - `30 2 * * * root /usr/bin/flock -w 3600 /var/lock/scriptorium-nightly-ops.lock /usr/local/bin/scriptorium-nightly-backup.sh >> /var/log/scriptorium-backup.log 2>&1`
- Main script:
  - `/usr/local/bin/scriptorium-nightly-backup.sh`
- Scope:
  - Scriptorium stack backup streams only
- Main source paths backed up:
  - `/srv/scriptorium-stack/forgejo/data`
  - `/srv/scriptorium-stack/woodpecker/server-data`
  - Postgres logical dumps from Docker container `forgejo-postgres`
- Main backup output area:
  - `/srv/backups/scriptorium`
- Backup artifacts created/rotated there:
  - `forgejo-data-latest.tar.gz`
  - `woodpecker-data-latest.tar.gz`
  - `postgres-forgejo-latest.sql.gz`
  - `postgres-woodpecker-latest.sql.gz`
- Additional retention/prune targets in the same backup area:
  - `scriptoriumai-static-*.tgz`
  - `cutover-*`
- Workstation snapshot retention also pruned there if uploaded:
- `/srv/backups/scriptorium/workstation-snapshots`
  - this path is **not** used by the current Authorship automation design
- What it does:
  - performs preflight disk-capacity check
  - writes Scriptorium DB/data backups
  - prunes backup streams according to keep-latest policy

#### D. Nightly maintenance prune

- Schedule source:
  - `/etc/cron.d/scriptorium-maintenance`
- Current cadence:
  - nightly at `02:35`
- Cron entry:
  - `35 2 * * * root /usr/bin/flock -w 3600 /var/lock/scriptorium-nightly-ops.lock /usr/local/bin/scriptorium-maintenance-prune.sh nightly >> /var/log/scriptorium-maintenance.log 2>&1`
- Main script:
  - `/usr/local/bin/scriptorium-maintenance-prune.sh`
- Scope:
  - mixed
  - mostly Scriptorium ops maintenance, but some actions are broader host maintenance
- Main files/paths involved:
  - `/srv/scriptorium-stack/woodpecker/server-data/logs`
  - `/etc/logrotate.d/scriptorium-ops`
  - `/var/log/scriptorium-maintenance.log`
- Broader host-level actions:
  - Docker container prune
  - Docker image prune
  - Docker builder prune
  - journald vacuum
- What it does:
  - safely reclaims Docker/cache/log space
  - prunes old Woodpecker server logs
  - rotates/cleans ops logs

#### E. Shared notification helper

- Helper script:
  - `/usr/local/bin/scriptorium-ops-notify.sh`
- Config file:
  - `/etc/scriptorium/ops-alerts.env`
- Used by:
  - disk guard
  - Forgejo health guard
  - nightly backup preflight/failure paths
  - maintenance failure path

### 16.3 The Authorship Mirror Lane: What It Is And What It Is Not

The source path discussed here is:

- `C:\Users\Anthracite Ace\Desktop\Authorship`

That path is on the Windows workstation, not on the VPS.

So the correct implementation model is:

- workstation-side scheduled task
- scan the visible folder on Windows
- exclude `.git` storage/history from the mirror payload
- exclude `.zip` backup archives from the mirror payload
- package only new or modified visible non-zip files since the last successful mirror manifest
- upload those changed/new files to a fixed directory on the VPS

Helper scripts for that lane:

- `C:\ScriptoriumAI\ops\ssh\run-authorship-mirror-sync.ps1`
- `C:\ScriptoriumAI\ops\ssh\install-authorship-mirror-sync-task.ps1`

Current workstation-side staging note:

- the sync script now uses `%LOCALAPPDATA%\am` as its temporary staging root
- that local staging area is scratch space only, not the authoritative backup target
- the authoritative persisted mirror for this lane is the VPS path under `/srv/backups/scriptorium/authorship-current`
- old `C:\am` contents are legacy staging leftovers from the prior implementation and can be deleted manually when no mirror sync process is active
- stale local `%LOCALAPPDATA%\am` artifacts from aborted runs are janitor-cleaned automatically on later runs after the age threshold
- stale remote `/tmp/authorship-mirror-*` directories from aborted runs are also janitor-cleaned automatically on later runs after the age threshold
- the script now takes a single-instance lock so overlapping manual and scheduled mirror runs are rejected instead of racing the same staging area

Important design boundary:

- this is not a Git history lane
- it does not mirror `.git` or `.git/lfs` into the VPS backup target
- it does not mirror `.zip` backup archives into the VPS working-files target
- it stores the visible Authorship working files on the VPS and then adds changed/new non-zip files on later runs

Current VPS mirror target for the visible folder:

- `/srv/backups/scriptorium/authorship-current`

Current VPS manifest file for change detection:

- `/srv/backups/scriptorium/authorship-current/.authorship-sync-manifest.json`

### 16.4 Retroactive Vs Prospective Behavior For Authorship Mirror Sync

The Authorship mirror lane is **prospective from the time the workstation task is installed and begins running**.

That means:

- the first successful run seeds the VPS mirror with the current visible non-zip folder state excluding `.git`
- later runs compare visible non-zip files against the last successful mirror manifest
- if the VPS mirror already has that non-zip visible-file baseline plus manifest, later runs only need to add missing/new/modified working files
- later runs upload only new or modified visible non-zip files
- the job does not upload Git history or LFS object storage
- the job does not backfill historical states that were never mirrored before

So in practical terms:

- future scheduled runs are light if daily authorship changes are small
- the large first run is expected only when the VPS mirror/manifest does not already reflect the current visible non-zip folder state
- this lane is about server-side file backup, not source-control history
- local scratch data may exist briefly during a run under `%LOCALAPPDATA%\am`, but successful runs remove that staging data in cleanup
- aborted-run leftovers are not relied upon for cleanup anymore; later runs perform a stale-temp janitor pass locally and on the VPS temp lane
- current implementation uploads changed/new files; it does not currently prune files removed on the workstation from the VPS mirror automatically

### 16.5 Scope Clarification: Whole VPS Or Scriptorium Only?

For quick operator recall:

- whole-VPS check:
  - disk guard measures `/`
- Scriptorium-only check:
  - Forgejo health guard
  - nightly backup content streams
- mixed scope:
  - nightly maintenance prune

### 16.6 HyperTwist And VectorShell Note

Current policy remains:

- do not use this Scriptorium VPS backup lane as a general warehouse for full HyperTwist and VectorShell development worktrees

Future production note:

- once HyperTwist and VectorShell move to production, add their production-side backup streams explicitly
- do not treat that as justification for copying their whole active dev trees into the Scriptorium VPS backup root now

### 16.7 Operator Summary

If someone needs the shortest correct reading:

- disk guard: checks overall VPS root disk pressure
- Forgejo health guard: checks only Scriptorium Forgejo stack health
- nightly backup: backs up Scriptorium data/DB streams
- nightly maintenance: cleans Docker/log pressure, partly host-wide
- Authorship mirror sync: workstation scheduled task that stores the visible Authorship folder on the VPS and later uploads changed/new files only, not a VPS cron reading a Windows path directly
- zip backup archives are intentionally excluded from that working-files mirror target

## 15.1 DNS and Domain Outcomes

Confirmed `A` records to VPS `212.227.22.66`:
- `app.scriptoriumai.io`
- `api.scriptoriumai.io`
- `verticaltension.com`
- `app.verticaltension.com`
- `api.verticaltension.com`

IONOS behavior observed:
- Updating apex `@` to VPS can disable managed Webhosting records (`@`, `www`, `_dep_ws_mutex*`).
- Keep mail records (`MX`, `SPF`, `DKIM`, `DMARC`) unless explicitly migrating mail.

## 15.2 New Runtime Components Provisioned

Scriptorium bootstrap services:
- `scriptorium-app` on `127.0.0.1:8080`
- `scriptorium-api` on `127.0.0.1:8081`
- stack path: `/srv/scriptorium-app-api`

Vertical Tension API bootstrap:
- `verticaltension-api` on `127.0.0.1:8100`
- stack path: `/srv/verticaltension-api`

Vertical Tension web:
- final serving model is static files from `/var/www/verticaltension`
- temporary bootstrap `verticaltension-web` container on `127.0.0.1:8102` was removed after static cutover.

## 15.3 NGINX + TLS Added

New NGINX site files:
- `/etc/nginx/sites-available/scriptorium-app-api.conf`
- `/etc/nginx/sites-available/verticaltension-api.conf` (already existed and was finalized)
- `/etc/nginx/sites-available/verticaltension-web.conf`

Certificates issued/installed:
- `app.scriptoriumai.io` + `api.scriptoriumai.io`
- `verticaltension.com` + `app.verticaltension.com`
- `api.verticaltension.com` (earlier in same session)

Certbot pattern used:

```bash
certbot --nginx -d app.scriptoriumai.io -d api.scriptoriumai.io --non-interactive --agree-tos --register-unsafely-without-email --redirect
certbot --nginx -d verticaltension.com -d app.verticaltension.com --non-interactive --agree-tos --register-unsafely-without-email --redirect
```

## 15.4 Vertical Tension Static Deployment Source + Commands

Local artifact source:
- `C:\VerticalTension\dist`

Commands used (Windows -> VPS):

```powershell
ssh -i "C:\ScriptoriumAI\ops\ssh\scriptorium_vps_ed25519" root@212.227.22.66 "mkdir -p /tmp/vt-dist /var/www/verticaltension && rm -rf /tmp/vt-dist/*"
scp -i "C:\ScriptoriumAI\ops\ssh\scriptorium_vps_ed25519" -r "C:\VerticalTension\dist\*" root@212.227.22.66:/tmp/vt-dist/
ssh -i "C:\ScriptoriumAI\ops\ssh\scriptorium_vps_ed25519" root@212.227.22.66 "rm -rf /var/www/verticaltension/* && cp -a /tmp/vt-dist/. /var/www/verticaltension/ && chown -R www-data:www-data /var/www/verticaltension && nginx -t && systemctl reload nginx"
```

## 15.5 Troubleshooting Patterns Resolved

1. Cert issued but not installed:
- Cause: no matching `server_name` block in NGINX.
- Fix: create/enable vhost first, then install/deploy cert.

2. `404` during cutover:
- Cause: domain reached VPS but no active matching vhost/upstream/static root yet.
- Fix: define vhost and reload NGINX.

3. TLS principal mismatch (`wrong principal`):
- Cause: domain served with a certificate for another host.
- Fix: issue/install certificate for that exact hostname set.

4. Render removed too early:
- Impact: endpoint unavailable until VPS endpoint finalized.
- Rule: decommission old provider only after VPS health checks pass.

## 15.6 Validation Snapshot (Post-Addendum)

- `https://app.scriptoriumai.io/healthz` -> `200`
- `https://api.scriptoriumai.io/healthz` -> `200`
- `https://verticaltension.com` -> static app from `/var/www/verticaltension`
- `https://app.verticaltension.com` -> static app from `/var/www/verticaltension`
- `https://api.verticaltension.com/healthz` -> `200`

Companion non-sensitive summary:
- `docs/ops/VPS_DOMAIN_CUTOVER_INSIGHTS_2026-03-01.md`

## 16. New target VPS profile and migration objective

Target server class:

- provider: IONOS
- product: `VPS 12-24-720`
- target OS: Ubuntu `24.04` Linux
- target role: replacement for the current Forgejo/Woodpecker control-plane VPS

Replacement intent:

- move the current Forgejo/Woodpecker stack from the existing VPS to the new VPS
- preserve the current control-plane behavior:
  - Forgejo
  - Woodpecker server
  - VPS-local ops-only Woodpecker agent
- preserve the current data and operational surfaces:
  - Forgejo Postgres data
  - Woodpecker Postgres data
  - Forgejo data directory
  - Woodpecker server-data directory
  - NGINX vhost configuration
  - certbot renewal behavior
  - backup scripts
  - cron jobs
  - ops guard scripts
- keep the old VPS alive until the new one is validated, because the user may
  delete the old VPS shortly after successful cutover

## 17. Recommended migration method

Recommended path:

- do a fresh Ubuntu `24.04` host bootstrap
- then perform an application-level stack migration
- do **not** do a blind whole-disk or whole-VM clone unless the provider gives a
  tested image-clone path and the operator explicitly prefers that risk

Why this is the preferred path:

- the current stack is already documented in terms of services, paths, backups,
  and cron entries
- application-level migration preserves the data that matters without dragging
  opaque old-host state into the new VPS
- a fresh host gives cleaner SSH, machine-id, package, firewall, and Docker
  state
- rollback is simpler because the old VPS can stay online until DNS and CI prove
  the new host is healthy

What to migrate, exactly:

- `/srv/scriptorium-stack/.env`
- `/srv/scriptorium-stack/docker-compose.yml`
- `/srv/scriptorium-stack/forgejo/data`
- `/srv/scriptorium-stack/woodpecker/server-data`
- current Forgejo Postgres dump
- current Woodpecker Postgres dump
- `/etc/nginx/sites-available/*.conf` that are still meant to live on the
  replacement host
- linked NGINX enablement under `/etc/nginx/sites-enabled/`
- `/usr/local/bin/scriptorium-nightly-backup.sh`
- `/usr/local/bin/scriptorium-disk-guard.sh`
- `/usr/local/bin/scriptorium-forgejo-health-guard.sh`
- `/etc/cron.d/scriptorium-nightly-backup`
- `/etc/cron.d/scriptorium-ops-guards`
- `/etc/cron.d/scriptorium-maintenance`
- `/etc/systemd/journald.conf.d/90-scriptorium-limits.conf`
- any alerting env under `/etc/scriptorium/ops-alerts.env`

What should be recreated on the new host rather than copied blindly:

- SSH host keys
- firewall rules
- certbot-issued certificates
- machine-specific `systemd` state
- package cache and Docker image cache

## 18. Pending operator input required before live execution

This section is the exact nameserver and access information still needed.

### 18.1 New VPS access

Required:

- new VPS public IPv4
- new VPS public IPv6, if present and intended to be used
- root SSH access method:
  - password only, or
  - existing public-key path to install, or
  - console-only bootstrap first
- whether the new VPS is already provisioned with Ubuntu `24.04`
- whether Plesk or any panel image was selected by IONOS

### 18.2 DNS and nameserver inputs

Required:

- exact list of hostnames that must move to the new VPS
- current authoritative DNS provider for those hostnames
- whether nameserver delegation itself is changing or only `A`/`AAAA` records
  are changing
- if nameserver delegation is changing:
  - the new authoritative nameserver hostnames
  - glue IPs, if any nameserver is in-bailiwick
- if only record cutover is changing:
  - the exact `A` and `AAAA` records to update
- whether `MX`, `TXT`, SPF, DKIM, DMARC, or unrelated records exist on the same
  zones and must remain untouched
- desired TTL lowering window before cutover

Current likely hostnames from the existing runbook that may need confirmation:

- `git.scriptoriumai.io`
- `ci.scriptoriumai.io`
- possibly also:
  - `app.scriptoriumai.io`
  - `api.scriptoriumai.io`
  - `verticaltension.com`
  - `app.verticaltension.com`
  - `api.verticaltension.com`

Operator confirmation is required. Do not assume all of the above still belong
on the replacement host.

### 18.3 Cutover preference

Required:

- confirm whether the migration target is:
  - control-plane only, or
  - control-plane plus all currently hosted app/API/static vhosts
- confirm whether the operator wants:
  - application-level stack migration only, or
  - a provider-side disk/image clone attempt first

Default recommendation if no special constraint is given:

- application-level stack migration only

### 18.4 Development-access preference

Required:

- desired Linux username for interactive development
- whether the same user should be used by:
  - JetBrains Rider Gateway backend sessions
  - Visual Studio remote Linux connections
- whether Docker access is needed for that user
- whether the repo should be cloned fresh on the new host or copied from the old
  host

Default recommendation:

- create a dedicated non-root development user
- give that user SSH key access
- add that user to the `docker` group only if container control is actually
  required
- keep root only for ops and cutover

## 19. Migration execution plan

### Phase A. Freeze and audit the old VPS

1. Confirm the old host is still healthy enough to migrate.
2. Export and verify:
   - Forgejo Postgres dump
   - Woodpecker Postgres dump
   - Forgejo data tarball
   - Woodpecker data tarball
3. Record current live values:
   - `docker compose ps`
   - `docker compose config`
   - NGINX site files
   - `/etc/cron.d/*scriptorium*`
   - `systemctl list-timers --all`
   - certbot status
4. Lower relevant DNS TTLs before cutover if the DNS provider allows it.

### Phase B. Bootstrap the new Ubuntu 24.04 VPS

1. Patch and harden the base OS.
2. Install the baseline packages:

```bash
sudo apt update
sudo apt install -y \
  openssh-server rsync curl unzip zip make gdb build-essential \
  nginx certbot python3-certbot-nginx \
  docker.io docker-compose-plugin \
  postgresql-client ca-certificates jq
```

3. Validate:
   - SSH reachable
   - Docker working
   - NGINX installed
   - no conflicting panel services
4. If the IONOS image included a panel, remove it before stack deployment.

### Phase C. Create the stack root and restore application data

1. Create `/srv/scriptorium-stack`.
2. Copy in:
   - `.env`
   - `docker-compose.yml`
3. Restore:
   - Forgejo data directory
   - Woodpecker server-data directory
4. Start the compose stack without DNS cutover first.
5. Restore Postgres data into the new stack if the compose/project bootstrap did
   not already materialize the live databases from copied volumes.

### Phase D. Recreate ops scripts and cron

Recreate and verify:

- `/usr/local/bin/scriptorium-nightly-backup.sh`
- `/usr/local/bin/scriptorium-disk-guard.sh`
- `/usr/local/bin/scriptorium-forgejo-health-guard.sh`
- `/etc/cron.d/scriptorium-nightly-backup`
- `/etc/cron.d/scriptorium-ops-guards`
- `/etc/cron.d/scriptorium-maintenance`
- `/etc/systemd/journald.conf.d/90-scriptorium-limits.conf`
- `/etc/scriptorium/ops-alerts.env`

Validation:

```bash
sed -n '1,20p' /etc/cron.d/scriptorium-nightly-backup
sed -n '1,20p' /etc/cron.d/scriptorium-ops-guards
sed -n '1,20p' /etc/cron.d/scriptorium-maintenance
systemctl list-timers --all --no-pager | grep -E 'certbot|scriptorium|e2scrub_all'
```

### Phase E. Recreate NGINX and certificates

1. Install the intended NGINX site files.
2. Validate config:

```bash
nginx -t
```

3. Only after DNS points to the new VPS, issue or renew certificates with
   `certbot --nginx`.
4. Do not copy old Let's Encrypt material as the primary plan; reissue on the
   new host instead.

### Phase F. Pre-cutover validation on the new VPS

Validate before touching DNS:

- containers healthy
- Forgejo reachable locally
- Woodpecker reachable locally
- gRPC loopback binding present on `127.0.0.1:9000`
- NGINX config valid
- backup scripts callable
- cron files present
- data counts match expectation

Minimum verification examples:

```bash
docker compose --env-file /srv/scriptorium-stack/.env -f /srv/scriptorium-stack/docker-compose.yml ps
ss -ltnp | grep -E '127.0.0.1:3000|127.0.0.1:8000|127.0.0.1:9000|:2222'
docker logs --tail 100 forgejo
docker logs --tail 100 woodpecker-server
docker logs --tail 100 woodpecker-agent
```

### Phase G. DNS or nameserver cutover

1. Change only the required records or nameserver delegation after the new VPS
   is already healthy.
2. Wait for propagation.
3. Reissue certbot if hostname validation depended on the cutover.
4. Confirm external reachability for every moved hostname.

### Phase H. Post-cutover validation

Minimum acceptance:

- Forgejo login works
- repo clone/fetch works over SSH
- Woodpecker UI loads
- webhook-triggered pipeline registration still works
- nightly cron files are installed
- backup outputs are written on the new host
- NGINX reload is clean

### Phase I. Old VPS retirement

Only retire the old VPS after:

- DNS has settled
- Forgejo and Woodpecker are healthy on the new host
- at least one real backup run completed on the new host
- at least one real Woodpecker pipeline and webhook round-trip succeeded
- Rider and Visual Studio remote development both connect successfully

## 20. JetBrains Rider Gateway remote-development setup

Source basis:

- JetBrains Rider 2026.1 official remote-development docs

Key official requirements:

- Linux backend host over SSH
- SFTP enabled
- recent AMD64 Linux, including Ubuntu
- JetBrains recommends at least `2` cores, `4 GB` RAM, and `5 GB` disk
- remote backend host needs outbound access to JetBrains download URLs unless
  the installer is uploaded manually
- the backend is self-contained and expects a `glibc`-based Linux environment

Recommended HyperTwist setup:

1. Create a non-root development user, for example `dev`.
2. Install the SSH key for that user.
3. Clone HyperTwist into a stable path such as:
   - `/home/dev/src/HyperTwist`
4. Ensure enough space exists under:
   - `/home/dev/.cache/JetBrains/RemoteDev/dist`
   or choose a custom install path inside Gateway.
5. If the server should not reach JetBrains URLs directly, use the Gateway
   option to upload the installer from the local workstation.

Recommended server-side prep:

```bash
sudo adduser dev
sudo usermod -aG docker dev
sudo mkdir -p /home/dev/src
sudo chown -R dev:dev /home/dev/src
```

Connection flow:

1. Open JetBrains Rider locally.
2. Use `Remote Development`.
3. Create a new SSH connection to the new VPS.
4. Point Rider Gateway at `/home/dev/src/HyperTwist`.
5. Choose the backend version and connect.

Important operational note:

- Rider Gateway is the true thin-client remote IDE flow here.
- It is appropriate for full remote indexing, editing, build, run, and debug on
  the Linux host.

## 21. Visual Studio Community 2022 remote Linux development setup

Source basis:

- Microsoft Learn official Visual Studio 2022 Linux-development docs

Important model distinction:

- Visual Studio Community 2022 does support Linux development over SSH
- but it is not the same remote thin-client model as JetBrains Gateway
- for C++/CMake it copies sources and builds remotely over SSH and `rsync`
- for .NET on Linux it supports SSH attach-debug scenarios

Official Linux-side requirements for the C++ path include:

- `openssh-server`
- `build-essential`
- `gdb`
- `rsync`
- `make`
- `zip`
- recent `cmake`

Recommended install on Ubuntu 24.04:

```bash
sudo apt update
sudo apt install -y openssh-server build-essential gdb rsync make zip cmake ninja-build
```

Recommended install for .NET SSH attach-debug support:

```bash
sudo apt install -y openssh-server unzip curl
```

If Linux-side .NET attach debugging is needed, also install the correct .NET
runtime for Ubuntu 24.04 and avoid single-file publish for debug targets.

Recommended Visual Studio path:

1. Install the Linux/C++ workload on the Windows machine.
2. Add the new VPS in:
   - `Tools > Options > Cross Platform > Connection Manager`
3. Use a dedicated non-root SSH user.
4. Open or create a CMake-based Linux target configuration.
5. Let Visual Studio copy sources remotely via `rsync`.
6. Keep the project on the VPS under the same development user path used by
   Rider where practical.

Recommended practical split:

- use Rider Gateway for full remote IDE work on the VPS
- use Visual Studio Community for:
  - remote Linux C++/CMake build/debug
  - .NET attach-debug over SSH when needed

## 22. Recommended answer to the storage-copy versus fresh-upload decision

Current recommendation:

- do not copy the entire old VPS storage device
- do not migrate by opaque filesystem image unless rollback pressure forces it
- instead:
  - bootstrap a clean Ubuntu 24.04 host
  - restore the Forgejo/Woodpecker stack and data explicitly
  - clone HyperTwist fresh for development paths
  - restore only the exact ops scripts, cron files, NGINX vhosts, and stack
    data that the new host truly needs

Reason:

- it is the cleanest path for a host that will replace the current VPS rather
  than coexist long-term
- it avoids carrying stale Docker, TLS, host-key, cache, and package state
- it makes later deletion of the old VPS safer because the replacement process
  is explicit and reproducible

## 23. Exact information request to the operator

Before live execution, collect and fill these values:

- new VPS public IPv4:
- new VPS public IPv6, if any:
- new VPS root access method:
- confirm Ubuntu `24.04` installed: yes/no
- any preinstalled panel or non-default stack on the new VPS:
- exact hostnames to move:
- DNS provider or nameserver owner:
- are nameservers changing, or only record targets:
- if nameservers are changing, new nameserver hostnames:
- if records are changing, exact `A` and `AAAA` values to publish:
- TTL lowering window approved:
- desired development username:
- should Docker access be granted to that user: yes/no
- should app/API/static vhosts also move with Forgejo/Woodpecker: yes/no
- preferred migration mode if different from the recommendation above:

Do not start destructive cutover or old-VPS retirement until this section is
fully answered.

## 24. Operator updates captured on 2026-05-29

New-server details now fixed:

- target IPv4: `212.227.13.220`
- root login: `root`
- root password supplied separately by the operator and must remain unchanged
- Ubuntu `24.04` confirmed installed

Direct inspection result:

- no active Plesk services found
- no installed Plesk packages found
- no reinstall is required just to remove a panel
- no global IPv6 address was visible during inspection, so do not publish
  `AAAA` cutover records for the new host yet

Current operational migration rule after live checks:

- source-control heads are safe
- current production artifacts on the old VPS are not proven identical to the
  currently visible local build outputs
- therefore exact production continuity should prefer copying current deployed
  content from the old VPS during cutover rather than rebuilding from local
  workspaces on the first pass
