Skip to main content

Monitoring, Grafana and Slack

Host and provisioning​

LIVE ON STAGING — scanners-grafana runs in scanners-prod on a private t4g.small EC2 instance, 10.0.10.5, with IMDSv2 and SSM Online. Its instance role can read the Loki bucket and the Grafana-admin/Slack-bot secrets and assume the three platform CloudWatch read roles. Secret values are never documentation. E:evidence-aws.md:93-95

NOT VERIFIED — applied provisioning. The defined bootstrap writes datasource, dashboard and alerting files under /opt/grafana/provisioning, mounted read-only into the container. The pack did not verify these branch files on the host. Defined datasources: Loki, CloudWatch set-nonprod, CloudWatch set-shared and CloudWatch iontwrks management, using matching read roles in eu-north-1. Defined dashboard UID: set-platform-health. E:evidence-aws.md:96

Slack connections​

MechanismConnection / channelStatus and evidence
Amazon Q Developer / Chatbotset-alerts-infra → #alerts-infraLIVE ON STAGING; shared SNS subscription and Slack configuration. E:evidence-aws.md:79-80
Amazon Q Developer / Chatbotset-alerts-deploys → #deploysLIVE ON STAGING. E:evidence-aws.md:79-80
Amazon Q Developer / Chatbotset-alerts-security → #alerts-securityLIVE ON STAGING. E:evidence-aws.md:79-80
Amazon Q Developer / Chatbotset-alerts-critical → #alerts-criticalLIVE ON STAGING. E:evidence-aws.md:79-80
Grafana Slack contact pointsslack-prod, slack-security, slack-critical, slack-infra, slack-deploysNOT VERIFIED live destinations: pack records IDs were not resolved to channel names. Defined routing labels are not verified Slack names. E:evidence-aws.md:96
Grafana SNS contact pointsns-heartbeat → set-grafana-heartbeat (definition)APPROVED, IN BUILD; topic absent from observed list. E:evidence-aws.md:82,96 I:infra/shared-services/heartbeat.tf:9-11
Scanners SNSscanners-alerts, scanners-alerts-critical, scanners-alerts-security, scanners-deploysLIVE ON STAGING topics/subscriptions; exact Slack configuration and channel mapping NOT VERIFIED. E:evidence-aws.md:81
Cloudflare notificationsU1 health / proposed O5 readiness checkNOT VERIFIED; no Cloudflare configuration or test notification in the pack. A:.set/work/PS.14.md:60

Solid arrows summarize observed resources and reported paths; dotted arrows are unverified or approved additions. This is not proof that every alert has fired end to end. E:evidence-aws.md:69-82,95-103

Grafana rule catalog​

Every supplied rule is listed below from its definition, including Scanners rules because they share the box. Exact rule titles retain their source identifiers; a legacy SET label refers to Binary Networks staging. NOT VERIFIED loaded state for every row. PS.14 additions are APPROVED, IN BUILD as a delivery scope, not live alerts. A:.set/work/decisions-2026-10-26.md:66-72 E:evidence-aws.md:96

Signal/threshold summaries and operator actions are transcribed from rule annotations. Where no action is defined, the gap is explicit. Routing is a label/contact-point route, not proof of a Slack channel. No-data and pending duration are included because silence is a failure only for some rules.

rules.yaml​

NOT VERIFIED applied state.

RuleSignal / thresholdRoute labelTimingOperator actionDefinition
Scanners - Application ErrorsOne or more ERROR-level app log lines in the last 5 minutes. Query {job="scanners", level="ERROR"} in Grafana Explore for org/customer_id. Evaluator: gt 0.prodEvery 1m; pending 0s; no data OKSomething in the app logged an error in the last 5 minutes. This is the catch-all alert: it covers crashes, failed scans, bad API keys and startup problems. Open Grafana Explore, run the query, and read the error message to see what broke and which customer it affected.G:infra/grafana-provisioning/alerting/rules.yaml:9-61
Scanners - API Denial SpikeMore than 10 denied API calls in the last 5 minutes (possible probing or bad credentials). Query {job="scanners", outcome="denied"} for org/customer_id/ip. Evaluator: gt 10.securityEvery 1m; pending 0s; no data OKMore than 10 API requests were rejected in 5 minutes (wrong key, a blocked action, or the scan cooldown). A burst like this usually means someone is probing the API or a client is misconfigured. Check the source IP and key prefix in the logs.G:infra/grafana-provisioning/alerting/rules.yaml:62-114
Scanners - Scan Tool FailuresScan step failed: tool "{{ $labels.tool }}" exited abnormally (action="tool.failed"). Any tool-run crash fires this. Evaluator: gt 0.prodEvery 1m; pending 0s; no data OKA scanning tool ({{ $labels.tool }}) crashed or timed out in the last 10 minutes, so scans return fewer or no results until it is fixed. Query {job="scanners", action="tool.failed"} | json in Grafana Explore to read the tool, rc, and stderr (the reason) and which service/customer.G:infra/grafana-provisioning/alerting/rules.yaml:115-160
Scanners - Enrichment API DegradationMore than 10 enrichment-API errors in 10 minutes (NVD/EPSS/OSV/EOL). Severity scoring may be degraded. Query {job="scanners", action="enrich.api_error"}. Evaluator: gt 10.prodEvery 1m; pending 0s; no data OKThe outside data sources we use to score findings (NVD, EPSS, OSV, endoflife.date) are failing. Findings may get the wrong severity. This is usually a provider outage, a rate limit, or an expired API key. The logs show which provider failed.G:infra/grafana-provisioning/alerting/rules.yaml:161-200
Scanners - API Rate-Limit SpikeMore than 5 rate-limit (429) rejections in 5 minutes - aggressive client or load. Query {job="scanners", action="ratelimit.exceeded"}. Evaluator: gt 5.prodEvery 1m; pending 0s; no data OKClients hit the API rate limit (HTTP 429) more than 5 times in 5 minutes. Either one client is sending too many requests, or real traffic has grown. The logs show which key and path were throttled.G:infra/grafana-provisioning/alerting/rules.yaml:201-241
Scanners - Zero-Finding Scan SpikeMore than 3 full scans completed with zero findings in 30 minutes (possible silent degradation). Query {job="scanners", action="scan.zero_findings"}. Evaluator: gt 3.prodEvery 1m; pending 0s; no data OKMore than 3 full scans finished with nothing found in 30 minutes, even though they discovered assets. That can be normal for genuinely clean targets, but a burst usually means the scanners or enrichment are quietly broken. Spot-check one of those scans.G:infra/grafana-provisioning/alerting/rules.yaml:242-282
Scanners - Scan Failure SpikeMore than 3 scans failed in 15 minutes (systemic, not a one-off). Query {job="scanners", action="scan.failed"} for org/customer_id. Evaluator: gt 3.prodEvery 1m; pending 0s; no data OKMore than 3 scans failed in 15 minutes, which points to a systemic problem rather than a one-off. Look for a common cause such as a dependency being down, a bad deploy, or a queue issue in the failed-scan logs.G:infra/grafana-provisioning/alerting/rules.yaml:283-324
Scanners - API Service SilentThe api service has emitted no heartbeat for ~5 minutes (hung or crashed). Query {job="scanners", service="api"}. Evaluator: lt 1.criticalEvery 1m; pending 5m; no data AlertingThe API has gone completely silent for 5 minutes, with no logs at all. It is most likely hung or crashed, which means the public API may be down. This is page-worthy: check the api service immediately.G:infra/grafana-provisioning/alerting/rules.yaml:325-368
Scanners - Scan Stuck (worker died mid-run)A scan was failed by the sweeper because its worker stopped responding mid-run. Query {job="scanners", action="scan.stuck"} for org/job_id. Evaluator: gt 0.prodEvery 1m; pending 0s; no data OKThe API sweeper found a RUNNING scan whose worker went silent (died, OOM, or a scale-in race) and failed it. That scan was lost and must be re-run. If this recurs, check worker memory and ECS scale-in protection.G:infra/grafana-provisioning/alerting/rules.yaml:369-407
Scanners - Security Event (Key Rotated)An org.key_rotated event in the last 10 minutes. Query {job="scanners", action="org.key_rotated"} for org/customer_id. Evaluator: gt 0.securityEvery 1m; pending 0s; no data OKAn organization's API key was rotated in the last 10 minutes. This is usually a deliberate, normal action. But if nobody on the team did it, treat it as a possible compromise and confirm with the customer.G:infra/grafana-provisioning/alerting/rules.yaml:408-458
Scanners - Tool Inactivity-Kill EpidemicMore than 15 scan tools were killed for inactivity (hung) in 10 minutes. Query {job="scanners", action="tool.failed"} | json | killed="inactivity". Evaluator: gt 15.prodEvery 1m; pending 0s; no data OKA burst of scanning tools produced no output for the inactivity window and were killed as hung. A few is normal; a burst points to a network, egress, or tool problem hitting many scans at once.G:infra/grafana-provisioning/alerting/rules.yaml:459-498
Scanners - Credentials In LogsAWS session credentials or a security token appeared in a log line. Rotate immediately and find the logging site. Evaluator: gt 0.securityEvery 1m; pending 0s; no data OKA log line matched an AWS temporary-credential / session-token pattern. Secrets must never be logged. Treat as a potential exposure: rotate the affected credential and remove the logging that leaked it.G:infra/grafana-provisioning/alerting/rules.yaml:499-536

set-alerts.yaml​

NOT VERIFIED applied state.

RuleSignal / thresholdRoute labelTimingOperator actionDefinition
SET nonprod - no healthy API target (U2)HealthyHostCount. HealthyHostCount < 1 for 20 minutes. No data = the ALB/TG metrics are missing, which is not an outage by itself (Cloudflare U1 is the outage signal). Evaluator: lt 1.criticalEvery 10m; pending 10m; no data NoDataNo dedicated action recorded. Inspect the signal and escalate unexplained changes to Ron; do not alter code during response.G:infra/grafana-provisioning/alerting/set-alerts.yaml:20-53
SET nonprod - 5xx spike (E3/E4)HTTPCode_Target_5XX_Count, HTTPCode_ELB_5XX_Count, RequestCount. Target 5xx above 5% of requests (and at least 10), or at least 10 load-balancer 5xx (502/503/504), in 10 minutes. Check /set-nonprod/ecs and the ALB.prodEvery 10m; pending 0s; no data OKNo dedicated action recorded. Inspect the signal and escalate unexplained changes to Ron; do not alter code during response.G:infra/grafana-provisioning/alerting/set-alerts.yaml:54-125
SET nonprod - 4xx spike (E1)HTTPCode_Target_4XX_Count. At least 100 target 4xx in 10 minutes (the ALB has no per-code 401/403/404 metric): scanning or a broken client. Evaluator: gt 99.prodEvery 10m; pending 0s; no data OKNo dedicated action recorded. Inspect the signal and escalate unexplained changes to Ron; do not alter code during response.G:infra/grafana-provisioning/alerting/set-alerts.yaml:126-159
SET nonprod - database connections above 80% (C4/P3)DatabaseConnections. DatabaseConnections above 164 (80% of max_connections ~205, PS.10 T-C4 re-reads the real number). Check the API pools and the migrate task. Evaluator: gt 164.infraEvery 10m; pending 0s; no data NoDataNo dedicated action recorded. Inspect the signal and escalate unexplained changes to Ron; do not alter code during response.G:infra/grafana-provisioning/alerting/set-alerts.yaml:160-193
SET - alerting changed (D14b, independent route)AlertingChanged. A write to the alerting (set-alerts-* rules, the bus policy, set-alerts-critical, Amazon Q, the canary schedule) in some account. Check the D14a message and CloudTrail. Evaluator: gt 0.criticalEvery 10m; pending 0s; no data OKNo dedicated action recorded. Inspect the signal and escalate unexplained changes to Ron; do not alter code during response.G:infra/grafana-provisioning/alerting/set-alerts.yaml:194-232
SET - org trail stopped delivering (D4d)IncomingLogEvents. No events reached /aws/cloudtrail/iontwrks-org-trail in 30 minutes: the audit trail, and every push alert, is blind. Check the trail status. Evaluator: lt 1.criticalEvery 30m; pending 0s; no data AlertingNo dedicated action recorded. Inspect the signal and escalate unexplained changes to Ron; do not alter code during response.G:infra/grafana-provisioning/alerting/set-alerts.yaml:233-266
SET nonprod - alert publish retrying (H2)RetryInvocationAttempts. EventBridge in nonprod is retrying deliveries: the cross-account publish to set-alerts-critical is failing. Check role set-alerts-publish and the topic policy. Evaluator: gt 0.criticalEvery 30m; pending 0s; no data OKNo dedicated action recorded. Inspect the signal and escalate unexplained changes to Ron; do not alter code during response.G:infra/grafana-provisioning/alerting/set-alerts.yaml:267-300
SET management - alert publish retrying (H2)RetryInvocationAttempts. EventBridge in management is retrying deliveries: the cross-account publish to set-alerts-critical is failing. Check role set-alerts-publish and the topic policy. Evaluator: gt 0.criticalEvery 30m; pending 0s; no data OKNo dedicated action recorded. Inspect the signal and escalate unexplained changes to Ron; do not alter code during response.G:infra/grafana-provisioning/alerting/set-alerts.yaml:301-334
SET - hub delivery to Amazon Q failing (H3)NumberOfNotificationsFailed. A set-alerts-* topic (critical, infra, security, deploys) failed to deliver to Amazon Q. Check its channel configuration set-alerts-<channel>.criticalEvery 30m; pending 0s; no data OKNo dedicated action recorded. Inspect the signal and escalate unexplained changes to Ron; do not alter code during response.G:infra/grafana-provisioning/alerting/set-alerts.yaml:335-430
SET nonprod - database disk under 3 GiB free (S3)FreeStorageSpace. FreeStorageSpace under 3 GiB (absolute). Raise max_allocated_storage or find what grew. Evaluator: lt 3221225472.infraEvery 1h; pending 0s; no data NoDataNo dedicated action recorded. Inspect the signal and escalate unexplained changes to Ron; do not alter code during response.G:infra/grafana-provisioning/alerting/set-alerts.yaml:431-464
SET nonprod - no database backup in 26 hours (S2)BackupFinished. RDS did not report Finished DB Instance backup (RDS-EVENT-0002) for set-nonprod-db1 in 26 hours. Check automated backups. Evaluator: lt 1.infraEvery 1h; pending 0s; no data AlertingNo dedicated action recorded. Inspect the signal and escalate unexplained changes to Ron; do not alter code during response.G:infra/grafana-provisioning/alerting/set-alerts.yaml:465-501
SET nonprod - session role mixes portal and staff permissions (P1.17)SessionClaimMixed. A staging user's WorkOS role mixes portal and staff permissions, so none were granted. Find the user: /set-nonprod/ecs, streams api/*, filter { $.msg = "session claim mixes portal and staff permissions; none granted" } (fields organizationId, userId). Fix the role in WorkOS. Evaluator: gt 0.securityEvery 1h; pending 0s; no data OKNo dedicated action recorded. Inspect the signal and escalate unexplained changes to Ron; do not alter code during response.G:infra/grafana-provisioning/alerting/set-alerts.yaml:502-540
SET nonprod - daily spend above $6 (D13)EstimatedCharges. Estimated charges for 661255175401 rose more than $6 in 24 hours (normal ~$4.31/day). Check Cost Explorer. No data = billing alerts are not enabled in the management account.criticalEvery 6h; pending 0s; no data NoDataNo dedicated action recorded. Inspect the signal and escalate unexplained changes to Ron; do not alter code during response.G:infra/grafana-provisioning/alerting/set-alerts.yaml:541-577
SET nonprod - weekly canary missing (H1)Invocations. No set-alerts-canary invocation in nonprod for 8 days: the trail, the us-east-1 forwarder, the rule or the publish role is broken. Evaluator: lt 1.criticalEvery 6h; pending 0s; no data AlertingNo dedicated action recorded. Inspect the signal and escalate unexplained changes to Ron; do not alter code during response.G:infra/grafana-provisioning/alerting/set-alerts.yaml:578-611
SET shared - weekly canary missing (H1)Invocations. No set-alerts-canary invocation in shared for 8 days: the trail, the us-east-1 forwarder, the rule or the publish role is broken. Evaluator: lt 1.criticalEvery 6h; pending 0s; no data AlertingNo dedicated action recorded. Inspect the signal and escalate unexplained changes to Ron; do not alter code during response.G:infra/grafana-provisioning/alerting/set-alerts.yaml:612-645
SET management - weekly canary missing (H1)Invocations. No set-alerts-canary invocation in management for 8 days: the trail, the us-east-1 forwarder, the rule or the publish role is broken. Evaluator: lt 1.criticalEvery 6h; pending 0s; no data AlertingNo dedicated action recorded. Inspect the signal and escalate unexplained changes to Ron; do not alter code during response.G:infra/grafana-provisioning/alerting/set-alerts.yaml:646-678

set-app-alerts.yaml​

NOT VERIFIED applied state.

RuleSignal / thresholdRoute labelTimingOperator actionDefinition
SET nonprod - O1 restore-point restore burstRestorePointOk. 10 or more restore-point restores answered 200 in 5 minutes (insider or stolen staff session reverting answers). Evaluator: gt 9.securityEvery 5m; pending 0s; no data OKActivity Log: who and which client. Legit bulk repair = ignore. Not expected: BLOCK + REVOKE, page Ron.G:infra/grafana-provisioning/alerting/set-app-alerts.yaml:16-69
SET nonprod - O2 answer history not keptAnswerHistoryNotKept. 5 or more 'answer history not kept' in 1 hour (history is unrecoverable). Evaluator: gt 4.infraEvery 5m; pending 0s; no data OKCheck C4 and S3 panels (DB blip?). Nothing to rerun; if the DB was healthy page Ron next day.G:infra/grafana-provisioning/alerting/set-app-alerts.yaml:70-123
SET nonprod - O3 preview/download server errorsFileServeErrors. 5 or more 5xx on GET /api/standards/files/:fileId in 10 minutes. Evaluator: gt 4.prodEvery 5m; pending 0s; no data OKAfter a deploy: ROLLBACK. Otherwise see O8 (object unavailable). A browser CSP refusal is not visible here: check preview by hand in QA.G:infra/grafana-provisioning/alerting/set-app-alerts.yaml:124-177
SET nonprod - O6 client file sessions failingAssumeFailed. 3 or more STS assume_failed in 5 minutes: file opens fail closed (503). Evaluator: gt 2.infraEvery 5m; pending 0s; no data OKRole trust and NAT EIP unchanged? After a Terraform apply: page Ron to apply the previous state. After a deploy: ROLLBACK.G:infra/grafana-provisioning/alerting/set-app-alerts.yaml:178-231
SET nonprod - O8 object unavailable (many)ObjectUnavailable. 5 or more object_unavailable in 5 minutes (S3 403 on rows that exist: bucket policy, KMS, endpoint, mover marker). Evaluator: gt 4.criticalEvery 5m; pending 0s; no data OKMany: ROLLBACK, then page Ron to inspect the last bucket-policy apply.G:infra/grafana-provisioning/alerting/set-app-alerts.yaml:232-285
SET nonprod - O8 object unavailable (any)ObjectUnavailable. At least 1 object_unavailable in 15 minutes. Evaluator: gt 0.infraEvery 5m; pending 0s; no data OKOne object: page Ron (restore the version per README). One manual S3 delete is a known false positive.G:infra/grafana-provisioning/alerting/set-app-alerts.yaml:286-339
SET nonprod - O11b extraction launch failedExtractStartFailed. extract start failed (RunTask rejected: no stopped-task event exists for it). Evaluator: gt 0.infraEvery 5m; pending 0s; no data OKPage Ron (IAM, subnet or quota). Files stay pending until fixed.G:infra/grafana-provisioning/alerting/set-app-alerts.yaml:340-393
SET nonprod - O13 expert spend vs capExpertCostMicros. AI expert spend in 24 hours reached 80 percent of the 10 USD daily cap. Evaluator: gt 7999999.infraEvery 5m; pending 0s; no data OKWait for the UTC roll or Ron raises the Terraform cap. Unexplained: KILL-EXPERT, page Ron.G:infra/grafana-provisioning/alerting/set-app-alerts.yaml:394-447
SET nonprod - O14 expert cap reachedExpertCapRefused. expert_cap_refused scope=global: the feature is off for everyone. Evaluator: gt 0.infraEvery 5m; pending 0s; no data OKAs O13. Scope=user shows on the dashboard only; a script suspected: BLOCK.G:infra/grafana-provisioning/alerting/set-app-alerts.yaml:448-501
SET nonprod - O15 expert failingExpertFailed. 5 or more expert asks ended model_failed or timeout in 5 minutes (the ALB sees 200). Evaluator: gt 4.infraEvery 5m; pending 0s; no data OKCheck O16. 30 minutes sustained: KILL-EXPERT, page Ron.G:infra/grafana-provisioning/alerting/set-app-alerts.yaml:502-555
SET nonprod - O16 Bedrock throttles/errorsInvocationThrottles, InvocationClientErrors. Bedrock throttles >= 3 or client errors >= 1 in 10 minutes.infraEvery 5m; pending 0s; no data OKClient error (AccessDenied = IAM or Marketplace): page Ron. Throttle: wait; constant: KILL-EXPERT.G:infra/grafana-provisioning/alerting/set-app-alerts.yaml:556-635
SET nonprod - O17 quote verification dropping answersExpertAsked, ExpertDropped. Over 30 percent of expert asks in the last hour dropped answers (at least 10 asks).prodEvery 5m; pending 0s; no data OKFind the standard from citedPages/ids in the log; Remove that file or deactivate that standard. Many standards: KILL-EXPERT.G:infra/grafana-provisioning/alerting/set-app-alerts.yaml:636-715
SET nonprod - O31 expert completion latencyExpertLatencyMs. Expert p95 completion time over 120 s (timeout is 150 s). Evaluator: gt 120000.infraEvery 5m; pending 0s; no data OKCheck O16; KILL-EXPERT if sustained.G:infra/grafana-provisioning/alerting/set-app-alerts.yaml:716-769
SET nonprod - O31b API latency (non-expert)TargetResponseTime. API p95 target response time over 2 s for 15 minutes. Evaluator: gt 2.prodEvery 5m; pending 0s; no data OKCheck C4 and O16; deploys cause brief spikes.G:infra/grafana-provisioning/alerting/set-app-alerts.yaml:770-824
SET nonprod - O23 role check did not runRoleCheckSkipped. The WorkOS role permission check did not run. Evaluator: gt 0.infraEvery 5m; pending 0s; no data OKRESTART the API service.G:infra/grafana-provisioning/alerting/set-app-alerts.yaml:825-878
SET nonprod - O24 WorkOS unreachableAuthDown. 5 or more sign-in/session/serving-unexpired-token warnings in 5 minutes: sign-in is down. Evaluator: gt 4.infraEvery 5m; pending 0s; no data OKWorkOS status page; nothing to do on our side. Page Ron if more than 30 minutes.G:infra/grafana-provisioning/alerting/set-app-alerts.yaml:879-932
SET nonprod - O25a audit compensation failedAuditCompensationFailed. WorkOS write kept, our audit row lost, undo failed (compensated=false): WorkOS and our record diverge. Evaluator: gt 0.securityEvery 5m; pending 0s; no data OKPage Ron to reconcile from the WorkOS audit log. Do not BLOCK anything automatically.G:infra/grafana-provisioning/alerting/set-app-alerts.yaml:933-986
SET nonprod - O25b audit outcome missingAuditOutcomeMissing. Removal without outcome row, MFA reset outcome unknown, or view-as/refresh audit row lost. Evaluator: gt 0.securityEvery 5m; pending 0s; no data OKCheck C4 and S3 (DB blip?); page Ron to reconcile the user.G:infra/grafana-provisioning/alerting/set-app-alerts.yaml:987-1040
SET nonprod - O26 cross-client acceptanceInviteCrossClient. 3 or more invitations refused: the acceptor belongs to another client (1 hour). Evaluator: gt 2.securityEvery 5m; pending 0s; no data OKRemove/deactivate the invitation if the user is unknown.G:infra/grafana-provisioning/alerting/set-app-alerts.yaml:1041-1094
SET nonprod - O27 uninvited sign-insUninvitedSignIn. 10 or more 'sign-in refused: no workspace' in 15 minutes. Evaluator: gt 9.securityEvery 5m; pending 0s; no data OKNone expected (new user or probing); WorkOS covers brute force.G:infra/grafana-provisioning/alerting/set-app-alerts.yaml:1095-1148
SET nonprod - O28 groups sync failingGroupsSyncFailed. 3 or more workos group sync failures in 1 hour. Evaluator: gt 2.infraEvery 5m; pending 0s; no data OKRERUN groups:sync (429 is a known cause).G:infra/grafana-provisioning/alerting/set-app-alerts.yaml:1149-1202
SET nonprod - O29 Redis errorsRedisErrors. 3 or more redis client errors in 5 minutes. Evaluator: gt 2.infraEvery 5m; pending 0s; no data OKCheck ElastiCache events; RESTART.G:infra/grafana-provisioning/alerting/set-app-alerts.yaml:1203-1256
SET nonprod - O34 rotation failedRotationFailed, Errors. A DB password rotation failed or the rotator Lambda errored (credentials stop rotating).infraEvery 5m; pending 0s; no data OKRERUN rotation for that secret (README rotate-secret); if it fails again page Ron.G:infra/grafana-provisioning/alerting/set-app-alerts.yaml:1257-1336
SET nonprod - O30 API CPU/memoryCPUUtilization, MemoryUtilization. API service CPU or memory above 85 percent for 15 minutes.infraEvery 5m; pending 0s; no data OKPage Ron to resize (Terraform). Deploys cause brief spikes.G:infra/grafana-provisioning/alerting/set-app-alerts.yaml:1337-1418
SET nonprod - O32 DB burst credits / memoryCPUCreditBalance, FreeableMemory. t4g.small CPU credit balance under 50 or freeable memory under 100 MB.infraEvery 5m; pending 0s; no data OKPage Ron. A migration is a known cause.G:infra/grafana-provisioning/alerting/set-app-alerts.yaml:1419-1498
WatchdogAlways firing on purpose. Its hourly SNS publish is the heartbeat an AWS alarm watches (O33); Slack never receives it. Evaluator: gt 0.heartbeatEvery 5m; pending 0s; no data AlertingIf the O33 alarm fires: open Grafana by SSM port-forward and restart the container.G:infra/grafana-provisioning/alerting/set-app-alerts.yaml:1499-1534
SET nonprod - O8b broker line missingFileServeOk, AssumeWindowDistinctClients. At least 5 file reads succeeded in 2 hours with no assume_window line: the broker line (and assume_burst) may be dead. Needs default_value 0 on both metric filters.infraEvery 30m; pending 0s; no data OKWrong image? RESTART or ROLLBACK.G:infra/grafana-provisioning/alerting/set-app-alerts.yaml:1535-1614
SET nonprod - O9 purge tripwirePurgeTripped. The purge mass-deletion guard tripped. Evaluator: gt 0.infraEvery 30m; pending 0s; no data OKRun --dry-run, read the counts, RERUN with --approve <runId>:<count> only if reasonable; else page Ron.G:infra/grafana-provisioning/alerting/set-app-alerts.yaml:1615-1668
SET nonprod - O10 mover blocking jobsMoverSkipped. At least 2 job runs skipped because a file move is running (48 hours). Evaluator: gt 1.infraEvery 30m; pending 0s; no data OKstop-task the filemove task if hung. Temporary: delete this rule at P2.23 step 6.G:infra/grafana-provisioning/alerting/set-app-alerts.yaml:1669-1722
SET nonprod - O22 role driftRoleDrift. WorkOS role permissions differ from the committed mapping. Evaluator: gt 0.securityEvery 30m; pending 0s; no data OKFix in the WorkOS Dashboard (log field drift lists it); the 6-hour recheck clears it, or RESTART.G:infra/grafana-provisioning/alerting/set-app-alerts.yaml:1723-1781
SET nonprod - O12 certificate expiryDaysToExpiry. ACM certificate under 21 days to expiry (auto-renew starts at 60 days: the validation CNAME is gone). Evaluator: lt 21.infraEvery 1h; pending 0s; no data AlertingPage Ron to restore the DNS validation record; calendar the Cloudflare AOP root CA expiry.G:infra/grafana-provisioning/alerting/set-app-alerts.yaml:1782-1836

set-app-heartbeats.yaml​

NOT VERIFIED applied state.

RuleSignal / thresholdRoute labelTimingOperator actionDefinition
SET nonprod - O21 log pipeline aliveIncomingLogEvents. No log events in /set-nonprod/ecs for 15 minutes: every log-based alert is blind. Evaluator: lt 1.criticalEvery 5m; pending 0s; no data AlertingRESTART; check the log group and KMS policy; page Ron.G:infra/grafana-provisioning/alerting/set-app-heartbeats.yaml:11-71
SET nonprod - O19 cleanup heartbeatJobDone. No cleanup job_done outcome=ok in 26 hours (a skipped run never counts). Evaluator: lt 1.infraEvery 30m; pending 0s; no data AlertingRERUN cleanup. Skipped: see O10.G:infra/grafana-provisioning/alerting/set-app-heartbeats.yaml:72-127
SET nonprod - O20 purge heartbeatJobDone. No purge job_done outcome=ok in 26 hours (a skipped run never counts). Evaluator: lt 1.infraEvery 30m; pending 0s; no data AlertingRERUN purge. Skipped: see O10.G:infra/grafana-provisioning/alerting/set-app-heartbeats.yaml:128-188
SET nonprod - O35 rotation overdueRotationSucceeded. No successful DB password rotation in 100 days (90-day schedule). Evaluator: lt 1.infraEvery 1h; pending 0s; no data AlertingRERUN rotation (README), then ROTATE log.G:infra/grafana-provisioning/alerting/set-app-heartbeats.yaml:189-244

set-nonprod-rotation.yaml​

NOT VERIFIED applied state.

RuleSignal / thresholdRoute labelTimingOperator actionDefinition
SET nonprod - DB password rotation outside the 90-day scheduleNonScheduledRotation. RotateSecret on {{ $labels.Secret }} by {{ $labels.Caller }}. Scheduled rotations never call it. Expected only for an incident rotation (README) or a new login's enrolment (set-nonprod-migrate-task). Anything else: treat as an incident. Evaluator: gt 0.criticalEvery 5m; pending 0s; no data OKDetails: CloudTrail (org trail) or the copy in /set-nonprod/ecs, account 661255175401, filter { $.detail.eventName = "RotateSecret" }.G:infra/grafana-provisioning/alerting/set-nonprod-rotation.yaml:20-58
SET nonprod - network interface removed with the rotation roleRotationRoleEniRemoval. {{ $labels.Call }} by {{ $labels.Caller }} with set-nonprod-db-rotation's credentials. Lambda's own ENI management and its DryRun checks are excluded, so this is the function's code or its stolen credentials. Treat as an incident. Evaluator: gt 0.criticalEvery 5m; pending 0s; no data OKDetails: CloudTrail (org trail) or the copy in /set-nonprod/ecs, account 661255175401, filter { $.source = "aws.ec2" }.G:infra/grafana-provisioning/alerting/set-nonprod-rotation.yaml:59-98
SET nonprod - rotation alert delivery failingFailedInvocations. EventBridge rule {{ $labels.RuleName }} could not deliver to /set-nonprod/ecs, so the rotation alerts are blind. Check the log group resource policy (rotation.tf). Evaluator: gt 0.criticalEvery 5m; pending 0s; no data OKNo dedicated action recorded. Inspect the signal and escalate unexplained changes to Ron; do not alter code during response.G:infra/grafana-provisioning/alerting/set-nonprod-rotation.yaml:99-132

Push detectors​

NOT VERIFIED exact loaded-rule equivalence. The pack verifies the set-alerts family and sample paths, not every pattern or threshold. Source definitions generate success/refusal pairs for CloudTrail; service events generate one rule. E:evidence-aws.md:75-76 I:infra/modules/security-alerts/main.tf:270-317

Generated rule nameSignalThresholdChannel suffixOperator actionDefinition
set-alerts-d1-ok, set-alerts-d1-refusedRoot used or root session issuedOne matching success/refusal eventsecurity, criticalExpected: never, except a deliberate break-glass by Ron. Otherwise: incident - check the IP and what the session did.I:infra/modules/security-alerts/main.tf:83-98
set-alerts-d2-okConsole sign-in without MFAOne matching success eventsecurityExpected: never. Find the user, force MFA, rotate its password.I:infra/modules/security-alerts/main.tf:99-114
set-alerts-d3-ok, set-alerts-d3-refusedIAM changedOne matching success/refusal eventsecurityNot OrganizationAccountAccessRole (Terraform), or refused: incident unless you know who did it.I:infra/modules/security-alerts/main.tf:115-124
set-alerts-d3-tf-ok[terraform] IAM changed by OrganizationAccountAccessRole - check it matches an apply you ranOne matching success eventinfraExpected during a Terraform apply. No apply running: incident - someone holds the admin role.I:infra/modules/security-alerts/main.tf:127-137
set-alerts-d4-ok, set-alerts-d4-refusedAudit trail or organisation membership tamperedOne matching success/refusal eventsecurity, criticalExpected: only a reviewed org-security apply. Otherwise: incident - check the trail is still logging.I:infra/modules/security-alerts/main.tf:138-154
set-alerts-d5-ok, set-alerts-d5-refusedKMS key disabled, deleted or its policy changedOne matching success/refusal eventsecurityExpected: a Terraform apply you ran. Otherwise: incident - cancel any scheduled deletion first.I:infra/modules/security-alerts/main.tf:155-163
set-alerts-d6-ok, set-alerts-d6-refusedS3 exposure or RDS snapshot sharedOne matching success/refusal eventsecurity, criticalExpected: a Terraform apply you ran. Otherwise: incident - check the bucket policy / snapshot attributes.I:infra/modules/security-alerts/main.tf:164-178
set-alerts-d7-ok, set-alerts-d7-refusedIngress opened to the internetOne matching success/refusal eventsecurityExpected: never (the ALB only admits Cloudflare). Revoke the rule, then find out who.I:infra/modules/security-alerts/main.tf:179-195
set-alerts-d9-ok, set-alerts-d9-refusedData or its logs deletedOne matching success/refusal eventsecurityExpected: a Terraform apply you ran. Otherwise: incident - restore from snapshot/version.I:infra/modules/security-alerts/main.tf:196-211
set-alerts-d11-ok, set-alerts-d11-refusedWrite outside eu-north-1One matching success/refusal eventsecurityBinary Networks staging runs only in eu-north-1: incident.I:infra/modules/security-alerts/main.tf:212-233
set-alerts-canaryWeekly canary - the alert path works, no actionOne matching eventinfraExpected every Monday 07:00 UTC from each account. A call other than us-east-1 sns CreateTopic: check the schedule set-alerts-canary.I:infra/modules/security-alerts/main.tf:236-248
set-alerts-d14-ok, set-alerts-d14-refusedAlerting changedOne matching success/refusal eventcriticalExpected: a PS.10 Terraform apply you ran. Otherwise: incident - the alerts may be blind; check CloudTrail.I:infra/modules/security-alerts/main.tf:249-267
set-alerts-d8a-ok, set-alerts-d8a-refusedDatabase login secret read by an unexpected identityOne matching success/refusal eventsecurityExpected readers are the API task, the migrate task and the rotator. Anyone else: incident - rotate the secret (README).I:infra/nonprod/alerts.tf:39-47
set-alerts-d8b-ok, set-alerts-d8b-refused[INCIDENT] owner credential readOne matching success/refusal eventsecurityOnly set-nonprod-migrate-exec reads it (once per migration). This was someone else, or a refused read: incident - rotate set/nonprod/db1/owner.I:infra/nonprod/alerts.tf:48-56
set-alerts-d8b-expected-ok[expected] owner credential read for a migrationOne matching success eventdeploysExpected once per migration (deploy) and once nightly at 03:00 UTC (the restore-point purge, purge.tf). Neither just now: incident.I:infra/nonprod/alerts.tf:57-73
set-alerts-d8c-ok, set-alerts-d8c-refusedBatchGetSecretValue calledOne matching success/refusal eventsecurityThe app never calls it. Incident: find which secrets the caller could read.I:infra/nonprod/alerts.tf:74-80
set-alerts-d8d-ok, set-alerts-d8d-refusedDatabase secret changedOne matching success/refusal eventsecurityExpected: a Terraform apply you ran (owner secret). Otherwise: incident.I:infra/nonprod/alerts.tf:83-96
set-alerts-d10-ok, set-alerts-d10-refusedDeploy path changedOne matching success/refusal eventsecurityExpected: a Terraform apply you ran. Anything else: incident - check which image the service runs.I:infra/nonprod/alerts.tf:99-110
set-alerts-d10-deploy-ok[deploy] by pipeline - check it matches a develop runOne matching success eventdeploysExpected: one per merge to develop (Deploy staging run in set-app). No matching run: incident - stolen deploy credentials.I:infra/nonprod/alerts.tf:111-125
set-alerts-r1-ok, set-alerts-r1-refusedDB password rotation outside the 90-day scheduleOne matching success/refusal eventsecurityNot the migrate task's enrolment (r1-enrol). Expected only for an incident rotation (README). Anything else: incident.I:infra/nonprod/alerts.tf:128-135
set-alerts-r1-enrol-okDB login enrolled for rotation by the migrate taskOne matching success eventinfraExpected once per new login (a migration created it). No migration ran: incident.I:infra/nonprod/alerts.tf:137-147
set-alerts-r2-ok, set-alerts-r2-refusedNetwork interface removed with the rotation roleOne matching success/refusal eventsecurityLambda's own ENI management and DryRun checks are excluded: this is the function's code or its stolen credentials. Incident.I:infra/nonprod/alerts.tf:148-157
set-alerts-u3API task crashedOne matching eventinfraSeveral in a row = crash loop. Check the service events and /set-nonprod/ecs (stream prefix api).I:infra/nonprod/alerts.tf:158-170
set-alerts-u4API deployment failed or rolled backOne matching eventcritical, deploysThe circuit breaker rolled back. Check the Deploy staging run and the service events.I:infra/nonprod/alerts.tf:171-184
set-alerts-u5Database migration failedOne matching eventcriticalRead /set-nonprod/ecs stream prefix migrate. The release did not complete.I:infra/nonprod/alerts.tf:185-202
set-alerts-u6Cleanup job failedOne matching eventinfraA Standards Library purge did not finish; objects of items removed over 30 days ago stay until a run succeeds (the next one retries). Read /set-nonprod/ecs stream prefix cleanup: one line per task, the error class only.I:infra/nonprod/alerts.tf:205-222
set-alerts-u7Nightly purge failed (restore points, answer files)One matching eventinfraRestore points older than a year, and stray or removed client answer files, stay until a run succeeds (the next night retries; expired points are already hidden from every read). A tripped tenant (too many files selected) also exits 1: read the counts, then use the reviewed --dry-run / --approve path. Read /set-nonprod/ecs stream prefix purge: one FAILED line per tenant.I:infra/nonprod/alerts.tf:225-242
set-alerts-u8Many client file sessions in five minutesOne matching eventinfraOne API task assumed file-access roles for 30 or more distinct clients within five minutes (a loop, a bug, or a bulk read). The count comes from the process it watches. Check /set-nonprod/ecs for the assume_window lines and CloudTrail AssumeRole events by SourceIdentity; if it is not a known job, treat as an incident and revoke sessions on set-nonprod-answers-s3.I:infra/nonprod/alerts.tf:246-261
set-alerts-u9Standards extraction failedOne matching eventinfraThat standard's file stays pending. Re-run the extract task for it (README Runbooks, RERUN); if it fails again, Remove the file (existing soft delete, restorable 30 days). Read /set-nonprod/ecs stream prefix extract.I:infra/nonprod/alerts.tf:264-281
set-alerts-s2pAutomated backups switched off on db1One matching eventinfraExpected: never. Re-enable the backup retention (data.tf) and find who changed it.I:infra/nonprod/alerts.tf:282-291
set-alerts-s3pdb1 storage nearly full, exhausted or autoscaling failedOne matching eventinfraCheck FreeStorageSpace; raise max_allocated_storage (data.tf) if autoscaling hit its ceiling.I:infra/nonprod/alerts.tf:292-300
set-alerts-u10Grafana stopped reportingOne matching eventcriticalNo Watchdog publish for 3 hours: Grafana or its box is down, so every Grafana alert is blind (push alerts still work). Open Grafana over SSM port-forward and restart the container (nonprod README, Runbooks, GRAFANA).I:infra/shared-services/alerts.tf:111-126
set-alerts-d10e-ok, set-alerts-d10e-refusedContainer registry changedOne matching success/refusal eventsecurityExpected: a Terraform apply you ran. Anything else: incident - check set-api's images and policy.I:infra/shared-services/alerts.tf:129-142
set-alerts-d12-ok, set-alerts-d12-refusedTerraform state touched by an unexpected identityOne matching success/refusal eventsecurityData event: not in CloudTrail event history; search the eventID in /aws/cloudtrail/iontwrks-org-trail (management account). Only OrganizationAccountAccessRole (Terraform) touches it: incident.I:infra/shared-services/alerts.tf:148-155
set-alerts-d12-ops-okTerraform state touched beyond what Terraform doesOne matching success eventsecurityData event: search the eventID in /aws/cloudtrail/iontwrks-org-trail (management account). The Terraform role deleted state, touched another key or made another call: incident unless you did it.I:infra/shared-services/alerts.tf:156-173

LIVE ON STAGING management alarms: iontwrks-assume_role_no_externalid, iontwrks-cloudtrail_tamper and iontwrks-root_usage; thresholds NOT VERIFIED. Inspect the matching trail event and confirm whether Ron authorized it; escalate unexpected security changes. E:evidence-aws.md:70

LIVE ON STAGING scanners push rules: scanners-aws-health, scanners-ecs-deployment, scanners-ecs-task-stopped, scanners-guardduty-findings. Exact patterns, thresholds, targets and channel mappings are NOT VERIFIED; the pack supplies names only. Obtain rule/target exports and a sanitized delivery test. E:evidence-aws.md:76,81

MERGED, NOT DEPLOYED GitHub workflow notifications use the named repository secret SLACK_WEBHOOK_URL (name only). Failed/timed-out named workflows trigger a message; weekday 07:00 UTC digest runs from the default branch. Channel and actual delivery are NOT VERIFIED. A:.github/workflows/alerts.yml:3-29,36-49 The Security reusable workflow also receives that secret by name. A:.github/workflows/security.yml:1-10

Heartbeats, health checks and conflicts​

APPROVED, IN BUILD — independent Grafana dead-man: Watchdog to SNS, then AWS NumberOfMessagesPublished below 1 for three hourly periods with missing data treated as breaching, routed through Amazon Q. Topic and alarm were not observed live. I:infra/shared-services/heartbeat.tf:9-50 E:evidence-aws.md:82

NOT VERIFIED — Cloudflare U1 and O5 configuration/delivery. O5 proposes readiness, but the deployment docs describe public readiness blocking. Verify an authenticated probe path before claiming it works. A:.set/work/PS.14.md:60 I:infra/nonprod/edge.tf:170-175,247-256

Source disagreement: PS.14's initial burst rows specify one-minute evaluation, whereas Ron selected five minutes for staging and the new rule group uses five minutes. This page records all three; none proves the loaded interval. A:.set/work/PS.14.md:48,62-65 A:.set/work/decisions-2026-10-26.md:68-70 G:infra/grafana-provisioning/alerting/set-app-alerts.yaml:1-14

The older C4 connection limit is explicitly estimated (164 as 80% of about 205). It is NOT VERIFIED as a measured database ceiling; verify max_connections and review after instance resizing. G:infra/grafana-provisioning/alerting/set-alerts.yaml:10-11

Later app-pack provisioning evidence​

LIVE ON STAGING for the observed bootstrap/history only: live instance user-data writes only datasources/loki.yaml and pins Grafana 11.1.0. SSM history records copying contact points, policies, set-alerts.yaml, set-nonprod-rotation.yaml and CloudWatch datasource files with restarts on October 22/24. That differs from the newer Terraform bootstrap described above. Terraform ignores user_data changes. NOT VERIFIED current loaded-rule state: a successful copy/restart is not a rule export. E:evidence.md:76-78

APPROVED, IN BUILD — PS.14 provisioning at fb394cd was never copied in the observed history. The new Watchdog, dashboard and feature rules are therefore not established live. E:evidence.md:77

LIVE ON STAGING push target mappings: u3 → infra; u4 → critical and deploys; u5 → critical; u6/u7/canary → infra. These map to the shared hub channels listed above. The current Terraform u7 title includes answer files, but the live purge task still runs only purge-restore-points.js without FILES_BUCKET. E:evidence.md:29,79

NOT VERIFIED Cloudflare routing, health checks, Grafana current rule export and Slack contact-point names. Verify through sanitized read-only exports and delivery observations. E:evidence.md:45,78