Observability and runbooks
LIVE ON STAGING — CloudWatch holds the API log group, four observed metric filters and zero nonprod CloudWatch alarms. Management has three alarms. The Grafana host and metric-read roles exist. NOT VERIFIED — which Grafana rules are loaded and successfully delivering. E:evidence-aws.md:68-76,93-96
The complete definition-level alert catalog lists each supplied Grafana rule and push detector with signal, threshold, route and action. A rule file is not evidence of an operational alert.
| Condition | Status | Operations-only response |
|---|---|---|
| No healthy API target / outage | NOT VERIFIED | Check service events and current task health, then restart or roll back to a known-good task revision. Escalate to Ron if unresolved. Cloudflare synthetic checks need independent verification. A:.set/work/PS.14.md:50,60 |
| Failed cleanup or purge | LIVE ON STAGING | Schedules are observed, successful runs are not. Inspect the relevant job log; rerun the existing task only after understanding the failure. E:evidence-aws.md:43-44 I:infra/nonprod/alerts.tf:185-241 |
| Missing job heartbeat | APPROVED, IN BUILD | Require a successful job_done outcome, not merely exit zero. Proposed missing-success window: 26 hours. A mover skip is not success. A:.set/work/PS.14.md:81-82 |
| Purge tripwire | APPROVED, IN BUILD | Read a dry-run report. Use the documented run/count approval only after reviewing the exact selection. Do not bypass a tripwire blindly. A:.set/work/PS.14.md:66 |
| Many client sessions | APPROVED, IN BUILD | Stop affected API/extraction workloads, revoke sessions and page Ron. Blocking one user cannot contain arbitrary code execution. A:.set/work/PS.14.md:63 |
| Expert spend or model failures | APPROVED, IN BUILD | Cap zero is the approved stop. A task revision must be released after the setting changes; Terraform alone does not update the service. Verify refused reservations and no model calls. No live drill is evidenced. A:.set/work/PS.14.md:50-52,72-78 |
| Role drift / audit gaps | APPROVED, IN BUILD | Review WorkOS configuration and vendor audit records; reconcile with Ron. Do not invent missing audit outcomes. A:.set/work/PS.14.md:84-89 |
| Grafana silence | APPROVED, IN BUILD | Independent SNS heartbeat alarm is defined. Use SSM to inspect/restart the Grafana container; the Amazon Q push path is separate. A:.set/work/PS.14.md:97-99 I:infra/shared-services/heartbeat.tf:9-50 |
These are procedures to use after verifying the named feature exists in the deployed release. No operation was executed while writing these docs. An on-call response must not require writing code or weakening isolation.
APPROVED, IN BUILD — Platform health dashboard, new feature alerts, boot-plus-six-hour role drift check and independent Grafana dead-man. Ron chose five-minute staging evaluation and a threshold review on 2026-11-03. Older PS.14 proposal rows say one minute for bursts; current provisioning says five minutes. A:.set/work/decisions-2026-10-26.md:66-72 G:infra/grafana-provisioning/alerting/set-app-alerts.yaml:1-14
NOT VERIFIED — heartbeat bootstrap, successful alert delivery, actual on-call channel membership, Cloudflare checks, dashboard data and a cap-zero drill. Verify with loaded rule exports, sanitized test notifications, observed successful job timestamps and app evidence. Do not call a dashboard green merely because a task exits zero.
The later app pack establishes copied Grafana files and restarts, not current rule loading. It also confirms PS.14 files were not copied and u3–u7/canary push destinations were read back. Use the monitoring page for those distinctions. E:evidence.md:76-79