Commit graph

1 commit

Author SHA1 Message Date
Cursor Agent
d84f0bf0dd
fix(api): retry DO lifecycle reset errors in ConnectorsWorkflow
Fixes SUPERMEMORY-BACKEND-JMN (17,259 events, 598 users, 111 days).

## Root cause

When Cloudflare resets a Durable Object isolate during an active
ConnectorsWorkflow sync run, the DO RPC returns an error with
`durableObjectReset: true` and/or the message "Aborting engine:
Grace period complete".  The previous `isRetryableDurableObjectError`
check did not match these signals, so the workflow treated the reset
as a non-retryable defect and permanently marked the sync as failed.

## Changes

### apps/api/src/lib/do-retry.ts — expanded matcher (core fix)
- Added `durableObjectReset: true` check on the error and its cause chain
- Added message checks for "Grace period complete", "destroyed",
  and "Network connection lost"
- Exported `DurableObjectResetError` typed class and
  `liftDurableObjectResetDefect` helper for use in the retry wrapper

### apps/api/src/workflows/connectors/steps/do-retry-wrapper.ts — new
- `withDORetry` wraps any Effect with 4 retries + exponential backoff
  (100 ms base, factor 2, jitter) for DO lifecycle reset errors
- Converts DO reset defects into typed `DurableObjectResetError`
  failures so the retry predicate can match them
- Mirrors the retry count/backoff of the existing mcpMountHandler
  pattern (out of scope for this fix)

### apps/api/src/workflows/connectors/plan-entitlement.ts — new
- `coerceConnectorWorkflowDefect` classifies DO reset errors as
  kind "do_lifecycle_reset" rather than "unhandled_defect" so
  Sentry and dashboards can distinguish exhausted retries from
  genuine programming errors

### apps/api/src/workflows/connectors/workflow.ts — new
- `runConnectorsWorkflow` wraps every DO-touching step with
  `withDORetry` so lifecycle resets are transparently retried
  before reaching the permanent-failure path

### Tests
- 27 unit tests for `isRetryableDurableObjectError`,
  `liftDurableObjectResetDefect`, and `DurableObjectResetError`
- 6 unit tests for `coerceConnectorWorkflowDefect`
- All 33 tests pass; `tsc --noEmit` is clean

Co-authored-by: Dhravya Shah <dhravya@supermemory.com>
2026-07-20 21:39:01 +00:00