Add public website crawl surfaces

docs/ops/HYPERTWIST_PUBLIC_WEBSITE_AUTH_BILLING_AND_DISTRIBUTION_PACKET_2026-06-22.md

@@ -179,6 +179,7 @@ for `hypertwist.app` when a built `website/dist/index.html` is present:
 - `WEBSITE_DIST_PATH` and `SERVE_STATIC_WEBSITE` now expose explicit deployment control for that lane
 - the runtime-readiness verifier now warns when same-origin public posture leaves that static-serving mode implicit
 - request-level server coverage now also proves the live route behavior for public/app routes versus `/api/*`, `/auth*`, `/health`, and missing asset paths
+- the frontend public deployment surface now also includes first-party `robots.txt` plus `sitemap.xml` assets for the current public route set, while keeping `/app`, `/login`, and `/register` out of crawler posture
 
 This is browser-based user access for the operator/account surface.
 

docs/v6_5_deep_manual_pack/HyperTwist/FEATURE_REGISTRY.md

@@ -263,7 +263,7 @@ repo.
 
 | Feature | Status | Primary authority | Notes |
 |---|---|---|---|
-| Public `hypertwist.app` marketing shell | Implemented now | first-party `website/` app + feature registry/roadmap authority | HyperTwist now has a dedicated first-party public web surface for homepage, about, resources, pricing, download, support, and legal routes. This lane is separate from the embedded Unreal browser runtime under `Content/Browser/` and does not claim browser-simulator parity. The same package now also carries a first-party external runtime-readiness verifier so deploy-time env and live health posture can be checked outside the dashboard, plus separated local-versus-production env templates whose placeholder values are intentionally rejected until real launch config is in place, bootstrap CI now validates both the frontend and auth-server website commands directly, and the auth server can now auto-serve the built `website/dist` bundle with bounded SPA fallback for same-origin public deployment. Request-level server coverage now also proves that public/app shell delivery does not shadow `/api/*`, `/auth*`, `/health`, or missing asset paths, while the pricing/download/notices routes now surface first-party preview-versus-launch posture from the same bounded launch checklist instead of relying on hidden operator-only status. The live website lane now also owns route-aware title/description/canonical/Open-Graph/Twitter metadata for the real `hypertwist.app` marketing surface so deployed public pages no longer remain on a single generic SPA title/description. The real `check-runtime-readiness` CLI is now also exercised against the checked-in production example env files, and a spawned `website/server` bootstrap proof now verifies the live same-origin process path from production-shaped env into `/health`, `/api/auth/health`, and built-shell serving. |
+| Public `hypertwist.app` marketing shell | Implemented now | first-party `website/` app + feature registry/roadmap authority | HyperTwist now has a dedicated first-party public web surface for homepage, about, resources, pricing, download, support, and legal routes. This lane is separate from the embedded Unreal browser runtime under `Content/Browser/` and does not claim browser-simulator parity. The same package now also carries a first-party external runtime-readiness verifier so deploy-time env and live health posture can be checked outside the dashboard, plus separated local-versus-production env templates whose placeholder values are intentionally rejected until real launch config is in place, bootstrap CI now validates both the frontend and auth-server website commands directly, and the auth server can now auto-serve the built `website/dist` bundle with bounded SPA fallback for same-origin public deployment. Request-level server coverage now also proves that public/app shell delivery does not shadow `/api/*`, `/auth*`, `/health`, or missing asset paths, while the pricing/download/notices routes now surface first-party preview-versus-launch posture from the same bounded launch checklist instead of relying on hidden operator-only status. The live website lane now also owns route-aware title/description/canonical/Open-Graph/Twitter metadata for the real `hypertwist.app` marketing surface so deployed public pages no longer remain on a single generic SPA title/description, plus first-party `robots.txt` and `sitemap.xml` assets for the public route set while keeping `/app`, `/login`, and `/register` out of crawler posture. The real `check-runtime-readiness` CLI is now also exercised against the checked-in production example env files, and a spawned `website/server` bootstrap proof now verifies the live same-origin process path from production-shaped env into `/health`, `/api/auth/health`, and built-shell serving. |
 | Browser-based operator/account dashboard | Implemented now | first-party `website/` app + shared auth/dashboard packet | A protected browser dashboard is now live for operator access, account state, download posture, browser-access boundary explanation, notices review, and bounded billing/entitlement status. It reuses the shared SuperTokens auth posture proven in FamiliarOS and ScriptoriumAI while remaining HyperTwist-specific in product content and boundary claims, the current auth-health surface now truthfully distinguishes configured versus reachable or ready shared-core posture while exposing fallback-active reason instead of hardcoding readiness, and the same dashboard now also surfaces launch-readiness truth for download URLs, checkout links, source/notices URLs, billing-secret/map configuration, and local-versus-public runtime deployment posture. Focused frontend coverage now also protects deep-link login redirect preservation, safe `next`-path normalization across auth entry points, fallback/email auth-bootstrap normalization, login/register continuation behavior, public download-gating behavior, protected-route/shell behavior, real lazy-route tree behavior for key public and protected paths, top-level app-bootstrap and SuperTokens-wrapper posture, login/register unhappy-path and OAuth-button behavior, support-topic fallback routing when live checkout is not configured, desktop-link verify-url/dashboard readiness behavior, and explicit `noindex,nofollow` posture on protected/auth browser surfaces. The validation lane now also has a bounded signed test-session harness under `TEST_MODE=testing` that proves `/api/auth/me` and `/api/auth/desktop-link` behavior through the live spawned auth-server process without widening production auth posture. |
 | Desktop download posture and browser-to-desktop pairing | Implemented now | first-party `website/` app + `website/server` desktop-link endpoints | Public download targets, dashboard-side release posture, and short-lived desktop-link token generation/verification are now first-party owned. The current server posture now enforces exact website-origin matching, bounded per-user issuance, one-time token consumption, and billing-backed plan/download entitlement resolution with focused `website/server` tests green on `2026-06-22`, and the verify handshake now returns the same resolved download-entitlement posture the dashboard sees instead of only identity plus plan/role. The public `/download` page now keeps raw download URLs behind the protected dashboard instead of exposing them directly, preserves requested platform continuity through `/app/downloads?platform=...`, and surfaces that requested target again after auth handoff inside the protected release lane. Both the public and protected download surfaces now also carry first-party rollout steps plus release/notices/source references so the desktop setup lane is more than a generic link bucket. Actual release URLs remain deployment configuration rather than hardcoded product truth. |
 | Paddle-ready pricing and billing webhook seam | Implemented now | first-party `website/` app + `website/server` billing endpoint | The public pricing surface now exists with plan structure, checkout-link configuration seams, and the same `/api/billing/paddle/webhook` endpoint family used by the broader product website lane. The current server now verifies `Paddle-Signature` against `PADDLE_WEBHOOK_SECRET` using the documented raw-body HMAC flow, persists a bounded first-party billing state file, and applies verified Paddle events into account/download entitlement state that the browser dashboard consumes, with focused `website/server` tests green on `2026-06-22`. A spawned live-process proof now also verifies that a real signed webhook updates processed-event health and persisted billing state through the actual auth-server runtime, not only helper-level store tests, and transaction events no longer leak their id into stored `subscriptionId` state. Production checkout URLs, secret management, and broader operator/admin billing workflows remain deployment/application tasks, not shipped-code omissions. |

docs/v6_5_deep_manual_pack/HyperTwist/ROADMAP.md

@@ -252,7 +252,9 @@ Current consolidated milestone snapshot:
   `noindex,nofollow`, plus richer first-party release guidance on both the
   public `/download` and protected `/app/downloads` surfaces so rollout steps,
   release references, and source/notices posture stay attached to the desktop
-  distribution lane itself,
+  distribution lane itself, plus first-party `robots.txt` and `sitemap.xml`
+  assets for the public route set while keeping `/app`, `/login`, and
+  `/register` out of crawler posture,
   persists a bounded first-party billing-state file, applies verified Paddle
   events into account/download entitlement state, and surfaces that resolved
   billing/download posture back through `/api/auth/me`, the protected browser

website/README.md

@@ -10,6 +10,7 @@ First-party `hypertwist.app` surface for HyperTwist:
 - dashboard-side launch-readiness surface for download, checkout, auth, and notice configuration
 - public launch-status callouts across pricing, download, and notices surfaces
 - route-aware metadata, canonical, Open Graph, Twitter-card, and protected-route noindex posture for the live `hypertwist.app` surface
+- first-party `robots.txt` plus `sitemap.xml` for the public crawlable route set while keeping `/app`, `/login`, and `/register` out of crawler posture
 - runtime diagnostics that distinguish local, mixed, and public auth deployment posture
 - public open-source notices surface required by HyperTwist's MPL distribution doctrine
 

website/public/robots.txt

@@ -0,0 +1,7 @@
+User-agent: *
+Allow: /
+Disallow: /app
+Disallow: /login
+Disallow: /register
+
+Sitemap: https://hypertwist.app/sitemap.xml

website/public/sitemap.xml

@@ -0,0 +1,39 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
+  <url>
+    <loc>https://hypertwist.app/</loc>
+  </url>
+  <url>
+    <loc>https://hypertwist.app/about</loc>
+  </url>
+  <url>
+    <loc>https://hypertwist.app/resources</loc>
+  </url>
+  <url>
+    <loc>https://hypertwist.app/docs</loc>
+  </url>
+  <url>
+    <loc>https://hypertwist.app/support</loc>
+  </url>
+  <url>
+    <loc>https://hypertwist.app/changelog</loc>
+  </url>
+  <url>
+    <loc>https://hypertwist.app/pricing</loc>
+  </url>
+  <url>
+    <loc>https://hypertwist.app/download</loc>
+  </url>
+  <url>
+    <loc>https://hypertwist.app/open-source-notices</loc>
+  </url>
+  <url>
+    <loc>https://hypertwist.app/privacy</loc>
+  </url>
+  <url>
+    <loc>https://hypertwist.app/terms</loc>
+  </url>
+  <url>
+    <loc>https://hypertwist.app/shipping-payment</loc>
+  </url>
+</urlset>