Notifications & Scheduled Jobs

The product's flows imply a steady stream of user-facing events — document awaiting review, questionnaire assigned, response needs revision, policy approved, report ready, member invited, vendor questionnaire assigned — and one business-critical scheduled job: a client's portal access auto-expires one year after creation (the "Annual License Expired" wall), the recurring-revenue lever. This page locks the notification subsystem and the scheduling mechanism behind it.

---

Part 1 — Notification system

Decision — ✅ Locked

Build the notification core in-house on primitives already in the stack: a per-tenant notifications model in Postgres, BullMQ for fan-out and retries, AWS SES for email, AWS SNS kept for internal ops alerts only. Real-time delivery to the React SPA is over Server-Sent Events (SSE), with TanStack Query polling as the fallback.

Why build (not buy)

• Data residency is the deciding constraint. S.E.T is compliance software; shipping tenant identity + notification content to a US-hosted notification SaaS is a hard sell for GRC buyers and conflicts with the EU/Israel residency posture set for WorkOS and Logz.io. Of the managed options, only Novu (self-host or EU/Frankfurt) clears the residency bar; Knock and Courier are US-hosted and ruled out despite Knock's superior product.
• We already own the primitives. SES (email), BullMQ/Redis (queue, retries, scheduling), Postgres (durable store), SNS (internal fanout). A managed platform mostly re-sells orchestration we already run.
• Volume economics. GRC notification events are human-paced and low-volume per tenant; per-message SaaS pricing buys little at this scale.
• Buy-fallback trigger: if the in-app inbox UI + preference center becomes a multi-sprint drag, adopt Novu self-hosted (MIT, EU-capable) for its <Inbox/> component and workflow engine while keeping SES as the email provider — the only "buy" that doesn't violate residency.

The pipeline

One domain event fans out to multiple channels, each with independent delivery status — the in-app row and the email are two channels of the same event, not two systems:

domain event ─► notification service ─► preference check (category × channel × frequency)
                                            │
                        ┌───────────────────┼───────────────────┐
                        ▼                   ▼                   ▼
                  in-app row          SES email           SNS (internal ops)
                 (notifications)   (notification_        (not the user inbox)
                                     deliveries)

Data model (per tenant schema)

See Data Model §M. Core tables:

Table	Purpose
notifications	Inbox rows: recipient_user_id, category (enum), title, body, data (jsonb deep-link/entity refs), read_at (nullable → unread if null), archived_at, created_at. Read/unread is a nullable timestamp, not a boolean (gives "read at" for free). Idempotency key prevents duplicate rows on retry. Index (recipient_user_id, read_at, created_at desc) for the unread badge + feed.
notification_deliveries	One row per channel attempt: notification_id, channel (in_app|email|sns), status (queued|sent|delivered|bounced|failed), provider_message_id (SES id), error, timestamps. Delivery status tracked independently per channel.
notification_preferences	One row per (user_id, category): channels (jsonb {in_app,email}), frequency (instant|daily_digest|weekly|off), quiet_hours_start/end + timezone. Checked before every send; tenant-level defaults can override user-level. Quiet hours defer (re-enqueue with delay), never drop.

Categories: document_review, questionnaire_assigned, response_revision, policy_approved, report_ready, license_expiring, member_invite, vendor_questionnaire.

Real-time transport: SSE, not WebSockets

The inbox is a one-directional, low-frequency server→client push — the textbook SSE case. SSE is a single HTTP connection with built-in auto-reconnect via the browser EventSource API; WebSockets add a bidirectional handshake we don't need (S.E.T has no chat/collab). Pattern:

• Durable store first, transport second — notifications persist to Postgres immediately; the inbox loads history via TanStack Query; SSE only pushes newly arrived items while the user is active.
• NestJS @Sse() endpoint; on a new notification, publish over Redis pub/sub so whichever Fargate task holds that user's SSE connection emits it. On the SSE event, TanStack Query invalidates or optimistically prepends.
• Fallback = polling — where corporate proxies buffer event streams (a real risk for enterprise customers), a 30–60s refetchInterval is an acceptable degradation for human-paced notifications.

---

Part 2 — Scheduled jobs & the annual-license lever

Decision — ✅ Locked

Amazon EventBridge Scheduler is the clock; a BullMQ job is the executor. A daily schedule triggers a scan job that finds licenses crossing each reminder milestone and expires lapsed ones. In-process cron (@nestjs/schedule) is rejected. Idempotency is enforced in Postgres with a unique (license_id, milestone) key.

Why not in-process cron

@nestjs/schedule runs inside one Node process — on Fargate it fires on every replica (duplicate sends) and doesn't survive task restarts/scale events. For a job that sends email and flips access state, that's the exact failure mode to avoid. EventBridge Scheduler is a managed, serverless clock with no daemon to drift or patch, independent of app uptime — AWS's own guidance steers container workloads off in-instance cron toward it. (BullMQ Job Schedulers — note the old "repeatable jobs" API is removed in v6 — are a valid Redis-native alternative since the schedule lives in Redis and fires once regardless of replica count; EventBridge is preferred for a revenue-critical job to keep the trigger independent of our own Redis/app.)

The daily flow

EventBridge Scheduler (daily) ─► enqueue BullMQ scan job ─► for each tenant schema:
    • for each milestone M in {d30,d14,d7,d1,expiry,grace1,grace3}:
        SELECT clients WHERE access_expires_at falls in M's window  ─► send reminder
    • UPDATE clients SET status='expired'
        WHERE access_expires_at < now() AND status='active'         ─► "expired" notice

Because the job is date-range based (not "who expires at this exact timestamp"), a skipped run self-heals on the next day's window. Configure retries + a DLQ on the schedule; prefer the "trigger a durable BullMQ job" path so BullMQ's retry/backoff owns execution reliability (EventBridge RunTask can transiently fail on Fargate capacity).

Idempotency (the crux)

BullMQ is at-least-once — a handler will occasionally run twice, so idempotency is mandatory:

• A license_reminder_log table with a unique constraint on (license_id, milestone). The sender does INSERT … ON CONFLICT DO NOTHING and only emails if the insert succeeded. A duplicated job hits the conflict and no-ops. Backed by the DB unique index (survives a Redis flush), not Redis alone.
• The expire action guards on state: UPDATE … SET status='expired' WHERE id=? AND status='active' — the affected-row count tells you whether you performed the transition, so the "expired" notification fires exactly once.

Reminder cadence & grace (the recurring-revenue lever)

When	Message	Purpose
T‑30d	"Your S.E.T portal renews in 30 days"	B2B procurement lead time
T‑14d	Value recap + one-click renewal link	Reduce friction
T‑7d	Action reminder, clear CTA	Primary conversion nudge
T‑1d	"Last chance before access pauses"	Urgency
Day 0	Access moves to the "Annual License Expired" wall; email confirms + renew CTA	The hard event
Expiry +1	"Access paused — reactivate in one click"	Grace begins
Expiry +3 / +7	Final dunning nudges	Recover before hard close

• Grace period 3–7 days; set state to grace/delinquent rather than deleting data — reactivation stays one click, history intact.
• Dunning spacing: 3–4 messages across ~15 days; more causes fatigue.
• Enterprise accounts requiring a formal PO: add an earlier T‑60/T‑90 touch for renewal runway.

Field signal

Independent engineering guidance favors SSE for notification feeds (vendor inbox SDKs lean WebSocket because it's their infra to manage). The notification-infra market has consolidated to Knock (best product, US-only), Novu (open-source, EU/self-host — the residency-safe pick), and Courier. AWS's container guidance actively steers teams off in-instance cron toward EventBridge Scheduler → ECS; BullMQ Job Schedulers is the Redis-native equivalent.

Cadence sources conflict by vertical (membership 30/14/3 vs SaaS 30/7/1 vs enterprise 90/60/30/7). The 30/14/7/1 + grace blend above is chosen for a B2B GRC buyer with procurement cycles.