Testing & QA Strategy

The Architecture commits to "automated cross-tenant isolation tests in CI on every deploy." This page defines the full test strategy around that — the layers, the tools, defensible coverage targets, how a schema-per-tenant compliance product proves isolation, how an LLM-heavy product tests non-deterministic output, and the CI gate that blocks a bad merge. Correctness and tenant isolation are paramount (SOC 2 / ISO 27001 / Amendment 13), and the SET scope budgeted zero QA hours — this page is the corrective.

Shape — the testing trophy, not the pyramid

For a TypeScript SaaS in 2025–2026 the center of gravity is integration tests: Vitest's fast cold start plus Testcontainers make "real dependency" tests cheap enough to be the default. Rough mix ≈ 70% unit / 20% integration / 10% E2E by count, but the value concentrates in the integration band.

Layer	Tooling	Covers for S.E.T
Static	tsc --noEmit, ESLint/Prettier, Zod parse-checks	Type safety; the shared Zod contracts across FE/BFF/API
Unit	Vitest + React Testing Library	Pure logic, components, scoring/policy functions
Integration/API	Vitest/Jest + Supertest + Testcontainers (real Postgres 16 + Redis)	NestJS resolvers/controllers on a real DB, Drizzle queries, BullMQ jobs, tenant isolation
Contract	Zod assertions; Pact-style (optional) for public REST	GraphQL↔BFF shapes, public REST stability, LLM JSON output shape
E2E	Playwright	Critical journeys, cross-browser, deploy smoke
A11y	@axe-core/playwright	WCAG 2.2 — an enterprise-sales requirement
Mutation	StrykerJS (targeted)	Test effectiveness on critical modules
Load	k6 (v1.0, native TS)	Dashboard GraphQL, report generation, evidence ingestion

Tool locks: Vitest (default runner; Jest acceptable on the NestJS side if avoiding two runners), Playwright over Cypress for new E2E, Supertest for HTTP, Testcontainers over mocks for the DB.

Coverage — measure the right thing

Raw line/branch % is a weak, gameable metric (100% coverage with zero assertions is possible). The position:

• Floor gate ≈ 70–80% line coverage as a ratchet that can't regress — not a target to maximize — with stricter branch coverage on critical paths: tenant isolation, auth/RBAC, policy scoring, report generation, evidence handling.
• What we actually gate on instead of raw %:
  • Critical-path coverage — enumerate the compliance-critical flows; each requires explicit tests.
  • Mutation score (StrykerJS) on high-risk modules only — proves tests would fail if the code were wrong. Gate on "no surviving mutants" for tenant-isolation, auth, and scoring code. Do not chase global mutation score.
  • Diff coverage on PRs — new/changed lines must be covered, rather than absolute repo %.

Cross-tenant isolation testing — the crown-jewel test

Runs in CI on every PR/deploy with Testcontainers giving a real Postgres 16 (mocks can't prove isolation — the isolation is Postgres behavior: search_path, schema grants, SET/RESET). Setup: spin PG 16, migrate ≥2 tenant schemas (tenant_a, tenant_b), seed distinguishable data. The leakage battery:

1. Positive isolation — request scoped to tenant A sees A's rows and exactly zero of B's.
2. search_path discipline — an unresolved/missing tenant → hard failure, never a silent fallback to public/a default schema.
3. Negative / cross-schema attempt — with search_path pinned to A, a schema-qualified read of tenant_b.* (or B's IDs passed into A-scoped resolvers) is denied/empty — catches IDOR at the data layer.
4. Connection-pool bleed — run tenant A then tenant B on the same pooled connection; assert search_path is reset per checkout/per job so B can't inherit A's. This is the subtle bug schema-per-tenant systems actually hit (RDS Proxy makes pool discipline non-negotiable).
5. Least privilege (SOC 2 strong) — if using per-tenant DB roles, assert a role cannot SET search_path to another tenant's schema — isolation at the grant level, not just app logic.

Testing the LLM features (questionnaires, policies, reports)

Layered cheapest-first — since AI output becomes audit evidence, grounding matters:

1. Contract/shape tests (deterministic, cheap, highest ROI) — validate every LLM output against its shared Zod schema; a malformed questionnaire/policy JSON fails the build regardless of prose quality.
2. Golden datasets + assertion evals — representative + tricky inputs with expected structured outcomes; assert on the deterministic parts ("policy references framework Y", "N sections present", required clauses present).
3. LLM-as-judge for subjective quality only — binary PASS/FAIL rubrics (not 1–5 Likert), versioned and aligned to human labels; reserved for what code can't check (on-topic, non-hallucinated).
4. Handle non-determinism statistically — run flaky cases N times (3–10), report pass-rate, evaluate the output not the reasoning trace.
5. Regression gating — golden + judge evals run before deploy; version prompts and re-run the eval set whenever a prompt or model version changes (silent prompt/model drift degrades quality without errors). Grow the golden set from real observed failures.

Zod + Vitest golden tests cover most of this; add a harness (DeepEval / Promptfoo) only when judge/eval volume grows.

Load / performance testing

k6 (v1.0, native TS, Go runtime, CI-friendly — fits the JS/TS team). Standardize on one tool. Load-test: dashboard aggregation GraphQL (p95/p99 under concurrent multi-tenant load), report generation (enqueue path + worker throughput under burst), evidence ingestion (sustained bulk write), LLM-backed endpoints (timeout/graceful-degradation when the model is slow). Run against nonprod only, synthetic tenants/data, off-hours; encode k6 thresholds as pass/fail gates so a perf regression can block a release.

CI gate — the ordered blocking checks

Fast checks run in parallel; a single gate job aggregates pass/fail; branch protection enforces green before merge; thresholds encoded as policy-as-code for change-control evidence.

1. Typecheck — tsc --noEmit
2. Lint / format — ESLint + Prettier (+ actionlint on workflows)
3. Unit — Vitest (+ diff-coverage floor, no regression)
4. Integration/API — Supertest + Testcontainers (real Postgres/Redis)
5. Cross-tenant isolation battery — hard gate (the compliance-critical one)
6. LLM contract/golden evals — Zod shape + golden assertions
7. SAST — CodeQL (push + weekly)
8. Dependency / SCA scan — Dependency Review / Snyk / Trivy; block vulnerable or license-noncompliant deps (supply-chain policy)
9. Build — app + container image
10. E2E smoke + a11y — Playwright critical-path + @axe-core/playwright (gate on critical + serious only)

Two-tier deploy (see Environments): nonprod auto-deploys on merge once gates pass, then runs E2E + isolation smoke against the deployed env; prod is gated on a required-reviewer approval from a green nonprod, runs smoke + isolation smoke post-deploy, rollback ready.

Test data: synthetic or sanitized data only in nonprod — never real customer data, ever (SOC 2 / ISO / Amendment 13). This is itself a documented, enforced control (test-data protection).

Field signal

The integration-heavy trophy is the reported 2026 default for TS SaaS (Vitest speed + Testcontainers). Cross-tenant isolation tests in CI on every PR are called out explicitly as multi-tenant best practice. LLM testing consensus: code-based evals for deterministic failures + binary LLM-as-judge + golden regression sets + N-run scoring. k6 is the modern default for JS/TS teams. GitHub Actions quality-gate + policy-as-code + branch protection + gated prod environments is the standard DevSecOps pattern.

Contested points, flagged: Playwright vs Cypress is near-consensus for Playwright but Cypress retains a minority following (low-risk either way). Vitest vs Jest on the NestJS side is a wash — pick one to avoid two-runner overhead. Coverage % figures in the literature are self-reported blog benchmarks — treat mutation score + critical-path coverage as the real quality signal, not any single %. Note also that schema-per-tenant is advised below ~200–500 tenants — beyond that it's a scaling bottleneck (Multi-Tenancy accepts this ceiling).