Monitoring, Grafana and Slack
Host and provisioning
LIVE ON STAGING — scanners-grafana runs in scanners-prod on a private t4g.small EC2 instance, 10.0.10.5, with IMDSv2 and SSM Online. Its instance role can read the Loki bucket and the Grafana-admin/Slack-bot secrets and assume the three platform CloudWatch read roles. Secret values are never documentation. E:evidence-aws.md:93-95
NOT VERIFIED — applied provisioning. The defined bootstrap writes datasource, dashboard and alerting files under /opt/grafana/provisioning, mounted read-only into the container. The pack did not verify these branch files on the host. Defined datasources: Loki, CloudWatch set-nonprod, CloudWatch set-shared and CloudWatch iontwrks management, using matching read roles in eu-north-1. Defined dashboard UID: set-platform-health. E:evidence-aws.md:96
Slack connections
| Mechanism | Connection / channel | Status and evidence |
|---|---|---|
| Amazon Q Developer / Chatbot | set-alerts-infra → #alerts-infra | LIVE ON STAGING; shared SNS subscription and Slack configuration. E:evidence-aws.md:79-80 |
| Amazon Q Developer / Chatbot | set-alerts-deploys → #deploys | LIVE ON STAGING. E:evidence-aws.md:79-80 |
| Amazon Q Developer / Chatbot | set-alerts-security → #alerts-security | LIVE ON STAGING. E:evidence-aws.md:79-80 |
| Amazon Q Developer / Chatbot | set-alerts-critical → #alerts-critical | LIVE ON STAGING. E:evidence-aws.md:79-80 |
| Grafana Slack contact points | slack-prod, slack-security, slack-critical, slack-infra, slack-deploys | NOT VERIFIED live destinations: pack records IDs were not resolved to channel names. Defined routing labels are not verified Slack names. E:evidence-aws.md:96 |
| Grafana SNS contact point | sns-heartbeat → set-grafana-heartbeat (definition) | APPROVED, IN BUILD; topic absent from observed list. E:evidence-aws.md:82,96 I:infra/shared-services/heartbeat.tf:9-11 |
| Scanners SNS | scanners-alerts, scanners-alerts-critical, scanners-alerts-security, scanners-deploys | LIVE ON STAGING topics/subscriptions; exact Slack configuration and channel mapping NOT VERIFIED. E:evidence-aws.md:81 |
| Cloudflare notifications | U1 health / proposed O5 readiness check | NOT VERIFIED; no Cloudflare configuration or test notification in the pack. A:.set/work/PS.14.md:60 |
Solid arrows summarize observed resources and reported paths; dotted arrows are unverified or approved additions. This is not proof that every alert has fired end to end. E:evidence-aws.md:69-82,95-103
Grafana rule catalog
Every supplied rule is listed below from its definition, including Scanners rules because they share the box. Exact rule titles retain their source identifiers; a legacy SET label refers to Binary Networks staging. NOT VERIFIED loaded state for every row. PS.14 additions are APPROVED, IN BUILD as a delivery scope, not live alerts. A:.set/work/decisions-2026-10-26.md:66-72 E:evidence-aws.md:96
Signal/threshold summaries and operator actions are transcribed from rule annotations. Where no action is defined, the gap is explicit. Routing is a label/contact-point route, not proof of a Slack channel. No-data and pending duration are included because silence is a failure only for some rules.
rules.yaml
NOT VERIFIED applied state.
| Rule | Signal / threshold | Route label | Timing | Operator action | Definition |
|---|---|---|---|---|---|
| Scanners - Application Errors | One or more ERROR-level app log lines in the last 5 minutes. Query {job="scanners", level="ERROR"} in Grafana Explore for org/customer_id. Evaluator: gt 0. | prod | Every 1m; pending 0s; no data OK | Something in the app logged an error in the last 5 minutes. This is the catch-all alert: it covers crashes, failed scans, bad API keys and startup problems. Open Grafana Explore, run the query, and read the error message to see what broke and which customer it affected. | G:infra/grafana-provisioning/alerting/rules.yaml:9-61 |
| Scanners - API Denial Spike | More than 10 denied API calls in the last 5 minutes (possible probing or bad credentials). Query {job="scanners", outcome="denied"} for org/customer_id/ip. Evaluator: gt 10. | security | Every 1m; pending 0s; no data OK | More than 10 API requests were rejected in 5 minutes (wrong key, a blocked action, or the scan cooldown). A burst like this usually means someone is probing the API or a client is misconfigured. Check the source IP and key prefix in the logs. | G:infra/grafana-provisioning/alerting/rules.yaml:62-114 |
| Scanners - Scan Tool Failures | Scan step failed: tool "{{ $labels.tool }}" exited abnormally (action="tool.failed"). Any tool-run crash fires this. Evaluator: gt 0. | prod | Every 1m; pending 0s; no data OK | A scanning tool ({{ $labels.tool }}) crashed or timed out in the last 10 minutes, so scans return fewer or no results until it is fixed. Query {job="scanners", action="tool.failed"} | json in Grafana Explore to read the tool, rc, and stderr (the reason) and which service/customer. | G:infra/grafana-provisioning/alerting/rules.yaml:115-160 |
| Scanners - Enrichment API Degradation | More than 10 enrichment-API errors in 10 minutes (NVD/EPSS/OSV/EOL). Severity scoring may be degraded. Query {job="scanners", action="enrich.api_error"}. Evaluator: gt 10. | prod | Every 1m; pending 0s; no data OK | The outside data sources we use to score findings (NVD, EPSS, OSV, endoflife.date) are failing. Findings may get the wrong severity. This is usually a provider outage, a rate limit, or an expired API key. The logs show which provider failed. | G:infra/grafana-provisioning/alerting/rules.yaml:161-200 |
| Scanners - API Rate-Limit Spike | More than 5 rate-limit (429) rejections in 5 minutes - aggressive client or load. Query {job="scanners", action="ratelimit.exceeded"}. Evaluator: gt 5. | prod | Every 1m; pending 0s; no data OK | Clients hit the API rate limit (HTTP 429) more than 5 times in 5 minutes. Either one client is sending too many requests, or real traffic has grown. The logs show which key and path were throttled. | G:infra/grafana-provisioning/alerting/rules.yaml:201-241 |
| Scanners - Zero-Finding Scan Spike | More than 3 full scans completed with zero findings in 30 minutes (possible silent degradation). Query {job="scanners", action="scan.zero_findings"}. Evaluator: gt 3. | prod | Every 1m; pending 0s; no data OK | More than 3 full scans finished with nothing found in 30 minutes, even though they discovered assets. That can be normal for genuinely clean targets, but a burst usually means the scanners or enrichment are quietly broken. Spot-check one of those scans. | G:infra/grafana-provisioning/alerting/rules.yaml:242-282 |
| Scanners - Scan Failure Spike | More than 3 scans failed in 15 minutes (systemic, not a one-off). Query {job="scanners", action="scan.failed"} for org/customer_id. Evaluator: gt 3. | prod | Every 1m; pending 0s; no data OK | More than 3 scans failed in 15 minutes, which points to a systemic problem rather than a one-off. Look for a common cause such as a dependency being down, a bad deploy, or a queue issue in the failed-scan logs. | G:infra/grafana-provisioning/alerting/rules.yaml:283-324 |
| Scanners - API Service Silent | The api service has emitted no heartbeat for ~5 minutes (hung or crashed). Query {job="scanners", service="api"}. Evaluator: lt 1. | critical | Every 1m; pending 5m; no data Alerting | The API has gone completely silent for 5 minutes, with no logs at all. It is most likely hung or crashed, which means the public API may be down. This is page-worthy: check the api service immediately. | G:infra/grafana-provisioning/alerting/rules.yaml:325-368 |
| Scanners - Scan Stuck (worker died mid-run) | A scan was failed by the sweeper because its worker stopped responding mid-run. Query {job="scanners", action="scan.stuck"} for org/job_id. Evaluator: gt 0. | prod | Every 1m; pending 0s; no data OK | The API sweeper found a RUNNING scan whose worker went silent (died, OOM, or a scale-in race) and failed it. That scan was lost and must be re-run. If this recurs, check worker memory and ECS scale-in protection. | G:infra/grafana-provisioning/alerting/rules.yaml:369-407 |
| Scanners - Security Event (Key Rotated) | An org.key_rotated event in the last 10 minutes. Query {job="scanners", action="org.key_rotated"} for org/customer_id. Evaluator: gt 0. | security | Every 1m; pending 0s; no data OK | An organization's API key was rotated in the last 10 minutes. This is usually a deliberate, normal action. But if nobody on the team did it, treat it as a possible compromise and confirm with the customer. | G:infra/grafana-provisioning/alerting/rules.yaml:408-458 |
| Scanners - Tool Inactivity-Kill Epidemic | More than 15 scan tools were killed for inactivity (hung) in 10 minutes. Query {job="scanners", action="tool.failed"} | json | killed="inactivity". Evaluator: gt 15. | prod | Every 1m; pending 0s; no data OK | A burst of scanning tools produced no output for the inactivity window and were killed as hung. A few is normal; a burst points to a network, egress, or tool problem hitting many scans at once. | G:infra/grafana-provisioning/alerting/rules.yaml:459-498 |
| Scanners - Credentials In Logs | AWS session credentials or a security token appeared in a log line. Rotate immediately and find the logging site. Evaluator: gt 0. | security | Every 1m; pending 0s; no data OK | A log line matched an AWS temporary-credential / session-token pattern. Secrets must never be logged. Treat as a potential exposure: rotate the affected credential and remove the logging that leaked it. | G:infra/grafana-provisioning/alerting/rules.yaml:499-536 |
set-alerts.yaml
NOT VERIFIED applied state.
| Rule | Signal / threshold | Route label | Timing | Operator action | Definition |
|---|---|---|---|---|---|
| SET nonprod - no healthy API target (U2) | HealthyHostCount. HealthyHostCount < 1 for 20 minutes. No data = the ALB/TG metrics are missing, which is not an outage by itself (Cloudflare U1 is the outage signal). Evaluator: lt 1. | critical | Every 10m; pending 10m; no data NoData | No dedicated action recorded. Inspect the signal and escalate unexplained changes to Ron; do not alter code during response. | G:infra/grafana-provisioning/alerting/set-alerts.yaml:20-53 |
| SET nonprod - 5xx spike (E3/E4) | HTTPCode_Target_5XX_Count, HTTPCode_ELB_5XX_Count, RequestCount. Target 5xx above 5% of requests (and at least 10), or at least 10 load-balancer 5xx (502/503/504), in 10 minutes. Check /set-nonprod/ecs and the ALB. | prod | Every 10m; pending 0s; no data OK | No dedicated action recorded. Inspect the signal and escalate unexplained changes to Ron; do not alter code during response. | G:infra/grafana-provisioning/alerting/set-alerts.yaml:54-125 |
| SET nonprod - 4xx spike (E1) | HTTPCode_Target_4XX_Count. At least 100 target 4xx in 10 minutes (the ALB has no per-code 401/403/404 metric): scanning or a broken client. Evaluator: gt 99. | prod | Every 10m; pending 0s; no data OK | No dedicated action recorded. Inspect the signal and escalate unexplained changes to Ron; do not alter code during response. | G:infra/grafana-provisioning/alerting/set-alerts.yaml:126-159 |
| SET nonprod - database connections above 80% (C4/P3) | DatabaseConnections. DatabaseConnections above 164 (80% of max_connections ~205, PS.10 T-C4 re-reads the real number). Check the API pools and the migrate task. Evaluator: gt 164. | infra | Every 10m; pending 0s; no data NoData | No dedicated action recorded. Inspect the signal and escalate unexplained changes to Ron; do not alter code during response. | G:infra/grafana-provisioning/alerting/set-alerts.yaml:160-193 |
| SET - alerting changed (D14b, independent route) | AlertingChanged. A write to the alerting (set-alerts-* rules, the bus policy, set-alerts-critical, Amazon Q, the canary schedule) in some account. Check the D14a message and CloudTrail. Evaluator: gt 0. | critical | Every 10m; pending 0s; no data OK | No dedicated action recorded. Inspect the signal and escalate unexplained changes to Ron; do not alter code during response. | G:infra/grafana-provisioning/alerting/set-alerts.yaml:194-232 |
| SET - org trail stopped delivering (D4d) | IncomingLogEvents. No events reached /aws/cloudtrail/iontwrks-org-trail in 30 minutes: the audit trail, and every push alert, is blind. Check the trail status. Evaluator: lt 1. | critical | Every 30m; pending 0s; no data Alerting | No dedicated action recorded. Inspect the signal and escalate unexplained changes to Ron; do not alter code during response. | G:infra/grafana-provisioning/alerting/set-alerts.yaml:233-266 |
| SET nonprod - alert publish retrying (H2) | RetryInvocationAttempts. EventBridge in nonprod is retrying deliveries: the cross-account publish to set-alerts-critical is failing. Check role set-alerts-publish and the topic policy. Evaluator: gt 0. | critical | Every 30m; pending 0s; no data OK | No dedicated action recorded. Inspect the signal and escalate unexplained changes to Ron; do not alter code during response. | G:infra/grafana-provisioning/alerting/set-alerts.yaml:267-300 |
| SET management - alert publish retrying (H2) | RetryInvocationAttempts. EventBridge in management is retrying deliveries: the cross-account publish to set-alerts-critical is failing. Check role set-alerts-publish and the topic policy. Evaluator: gt 0. | critical | Every 30m; pending 0s; no data OK | No dedicated action recorded. Inspect the signal and escalate unexplained changes to Ron; do not alter code during response. | G:infra/grafana-provisioning/alerting/set-alerts.yaml:301-334 |
| SET - hub delivery to Amazon Q failing (H3) | NumberOfNotificationsFailed. A set-alerts-* topic (critical, infra, security, deploys) failed to deliver to Amazon Q. Check its channel configuration set-alerts-<channel>. | critical | Every 30m; pending 0s; no data OK | No dedicated action recorded. Inspect the signal and escalate unexplained changes to Ron; do not alter code during response. | G:infra/grafana-provisioning/alerting/set-alerts.yaml:335-430 |
| SET nonprod - database disk under 3 GiB free (S3) | FreeStorageSpace. FreeStorageSpace under 3 GiB (absolute). Raise max_allocated_storage or find what grew. Evaluator: lt 3221225472. | infra | Every 1h; pending 0s; no data NoData | No dedicated action recorded. Inspect the signal and escalate unexplained changes to Ron; do not alter code during response. | G:infra/grafana-provisioning/alerting/set-alerts.yaml:431-464 |
| SET nonprod - no database backup in 26 hours (S2) | BackupFinished. RDS did not report Finished DB Instance backup (RDS-EVENT-0002) for set-nonprod-db1 in 26 hours. Check automated backups. Evaluator: lt 1. | infra | Every 1h; pending 0s; no data Alerting | No dedicated action recorded. Inspect the signal and escalate unexplained changes to Ron; do not alter code during response. | G:infra/grafana-provisioning/alerting/set-alerts.yaml:465-501 |
| SET nonprod - session role mixes portal and staff permissions (P1.17) | SessionClaimMixed. A staging user's WorkOS role mixes portal and staff permissions, so none were granted. Find the user: /set-nonprod/ecs, streams api/*, filter { $.msg = "session claim mixes portal and staff permissions; none granted" } (fields organizationId, userId). Fix the role in WorkOS. Evaluator: gt 0. | security | Every 1h; pending 0s; no data OK | No dedicated action recorded. Inspect the signal and escalate unexplained changes to Ron; do not alter code during response. | G:infra/grafana-provisioning/alerting/set-alerts.yaml:502-540 |
| SET nonprod - daily spend above $6 (D13) | EstimatedCharges. Estimated charges for 661255175401 rose more than $6 in 24 hours (normal ~$4.31/day). Check Cost Explorer. No data = billing alerts are not enabled in the management account. | critical | Every 6h; pending 0s; no data NoData | No dedicated action recorded. Inspect the signal and escalate unexplained changes to Ron; do not alter code during response. | G:infra/grafana-provisioning/alerting/set-alerts.yaml:541-577 |
| SET nonprod - weekly canary missing (H1) | Invocations. No set-alerts-canary invocation in nonprod for 8 days: the trail, the us-east-1 forwarder, the rule or the publish role is broken. Evaluator: lt 1. | critical | Every 6h; pending 0s; no data Alerting | No dedicated action recorded. Inspect the signal and escalate unexplained changes to Ron; do not alter code during response. | G:infra/grafana-provisioning/alerting/set-alerts.yaml:578-611 |
| SET shared - weekly canary missing (H1) | Invocations. No set-alerts-canary invocation in shared for 8 days: the trail, the us-east-1 forwarder, the rule or the publish role is broken. Evaluator: lt 1. | critical | Every 6h; pending 0s; no data Alerting | No dedicated action recorded. Inspect the signal and escalate unexplained changes to Ron; do not alter code during response. | G:infra/grafana-provisioning/alerting/set-alerts.yaml:612-645 |
| SET management - weekly canary missing (H1) | Invocations. No set-alerts-canary invocation in management for 8 days: the trail, the us-east-1 forwarder, the rule or the publish role is broken. Evaluator: lt 1. | critical | Every 6h; pending 0s; no data Alerting | No dedicated action recorded. Inspect the signal and escalate unexplained changes to Ron; do not alter code during response. | G:infra/grafana-provisioning/alerting/set-alerts.yaml:646-678 |
set-app-alerts.yaml
NOT VERIFIED applied state.
| Rule | Signal / threshold | Route label | Timing | Operator action | Definition |
|---|---|---|---|---|---|
| SET nonprod - O1 restore-point restore burst | RestorePointOk. 10 or more restore-point restores answered 200 in 5 minutes (insider or stolen staff session reverting answers). Evaluator: gt 9. | security | Every 5m; pending 0s; no data OK | Activity Log: who and which client. Legit bulk repair = ignore. Not expected: BLOCK + REVOKE, page Ron. | G:infra/grafana-provisioning/alerting/set-app-alerts.yaml:16-69 |
| SET nonprod - O2 answer history not kept | AnswerHistoryNotKept. 5 or more 'answer history not kept' in 1 hour (history is unrecoverable). Evaluator: gt 4. | infra | Every 5m; pending 0s; no data OK | Check C4 and S3 panels (DB blip?). Nothing to rerun; if the DB was healthy page Ron next day. | G:infra/grafana-provisioning/alerting/set-app-alerts.yaml:70-123 |
| SET nonprod - O3 preview/download server errors | FileServeErrors. 5 or more 5xx on GET /api/standards/files/:fileId in 10 minutes. Evaluator: gt 4. | prod | Every 5m; pending 0s; no data OK | After a deploy: ROLLBACK. Otherwise see O8 (object unavailable). A browser CSP refusal is not visible here: check preview by hand in QA. | G:infra/grafana-provisioning/alerting/set-app-alerts.yaml:124-177 |
| SET nonprod - O6 client file sessions failing | AssumeFailed. 3 or more STS assume_failed in 5 minutes: file opens fail closed (503). Evaluator: gt 2. | infra | Every 5m; pending 0s; no data OK | Role trust and NAT EIP unchanged? After a Terraform apply: page Ron to apply the previous state. After a deploy: ROLLBACK. | G:infra/grafana-provisioning/alerting/set-app-alerts.yaml:178-231 |
| SET nonprod - O8 object unavailable (many) | ObjectUnavailable. 5 or more object_unavailable in 5 minutes (S3 403 on rows that exist: bucket policy, KMS, endpoint, mover marker). Evaluator: gt 4. | critical | Every 5m; pending 0s; no data OK | Many: ROLLBACK, then page Ron to inspect the last bucket-policy apply. | G:infra/grafana-provisioning/alerting/set-app-alerts.yaml:232-285 |
| SET nonprod - O8 object unavailable (any) | ObjectUnavailable. At least 1 object_unavailable in 15 minutes. Evaluator: gt 0. | infra | Every 5m; pending 0s; no data OK | One object: page Ron (restore the version per README). One manual S3 delete is a known false positive. | G:infra/grafana-provisioning/alerting/set-app-alerts.yaml:286-339 |
| SET nonprod - O11b extraction launch failed | ExtractStartFailed. extract start failed (RunTask rejected: no stopped-task event exists for it). Evaluator: gt 0. | infra | Every 5m; pending 0s; no data OK | Page Ron (IAM, subnet or quota). Files stay pending until fixed. | G:infra/grafana-provisioning/alerting/set-app-alerts.yaml:340-393 |
| SET nonprod - O13 expert spend vs cap | ExpertCostMicros. AI expert spend in 24 hours reached 80 percent of the 10 USD daily cap. Evaluator: gt 7999999. | infra | Every 5m; pending 0s; no data OK | Wait for the UTC roll or Ron raises the Terraform cap. Unexplained: KILL-EXPERT, page Ron. | G:infra/grafana-provisioning/alerting/set-app-alerts.yaml:394-447 |
| SET nonprod - O14 expert cap reached | ExpertCapRefused. expert_cap_refused scope=global: the feature is off for everyone. Evaluator: gt 0. | infra | Every 5m; pending 0s; no data OK | As O13. Scope=user shows on the dashboard only; a script suspected: BLOCK. | G:infra/grafana-provisioning/alerting/set-app-alerts.yaml:448-501 |
| SET nonprod - O15 expert failing | ExpertFailed. 5 or more expert asks ended model_failed or timeout in 5 minutes (the ALB sees 200). Evaluator: gt 4. | infra | Every 5m; pending 0s; no data OK | Check O16. 30 minutes sustained: KILL-EXPERT, page Ron. | G:infra/grafana-provisioning/alerting/set-app-alerts.yaml:502-555 |
| SET nonprod - O16 Bedrock throttles/errors | InvocationThrottles, InvocationClientErrors. Bedrock throttles >= 3 or client errors >= 1 in 10 minutes. | infra | Every 5m; pending 0s; no data OK | Client error (AccessDenied = IAM or Marketplace): page Ron. Throttle: wait; constant: KILL-EXPERT. | G:infra/grafana-provisioning/alerting/set-app-alerts.yaml:556-635 |
| SET nonprod - O17 quote verification dropping answers | ExpertAsked, ExpertDropped. Over 30 percent of expert asks in the last hour dropped answers (at least 10 asks). | prod | Every 5m; pending 0s; no data OK | Find the standard from citedPages/ids in the log; Remove that file or deactivate that standard. Many standards: KILL-EXPERT. | G:infra/grafana-provisioning/alerting/set-app-alerts.yaml:636-715 |
| SET nonprod - O31 expert completion latency | ExpertLatencyMs. Expert p95 completion time over 120 s (timeout is 150 s). Evaluator: gt 120000. | infra | Every 5m; pending 0s; no data OK | Check O16; KILL-EXPERT if sustained. | G:infra/grafana-provisioning/alerting/set-app-alerts.yaml:716-769 |
| SET nonprod - O31b API latency (non-expert) | TargetResponseTime. API p95 target response time over 2 s for 15 minutes. Evaluator: gt 2. | prod | Every 5m; pending 0s; no data OK | Check C4 and O16; deploys cause brief spikes. | G:infra/grafana-provisioning/alerting/set-app-alerts.yaml:770-824 |
| SET nonprod - O23 role check did not run | RoleCheckSkipped. The WorkOS role permission check did not run. Evaluator: gt 0. | infra | Every 5m; pending 0s; no data OK | RESTART the API service. | G:infra/grafana-provisioning/alerting/set-app-alerts.yaml:825-878 |
| SET nonprod - O24 WorkOS unreachable | AuthDown. 5 or more sign-in/session/serving-unexpired-token warnings in 5 minutes: sign-in is down. Evaluator: gt 4. | infra | Every 5m; pending 0s; no data OK | WorkOS status page; nothing to do on our side. Page Ron if more than 30 minutes. | G:infra/grafana-provisioning/alerting/set-app-alerts.yaml:879-932 |
| SET nonprod - O25a audit compensation failed | AuditCompensationFailed. WorkOS write kept, our audit row lost, undo failed (compensated=false): WorkOS and our record diverge. Evaluator: gt 0. | security | Every 5m; pending 0s; no data OK | Page Ron to reconcile from the WorkOS audit log. Do not BLOCK anything automatically. | G:infra/grafana-provisioning/alerting/set-app-alerts.yaml:933-986 |
| SET nonprod - O25b audit outcome missing | AuditOutcomeMissing. Removal without outcome row, MFA reset outcome unknown, or view-as/refresh audit row lost. Evaluator: gt 0. | security | Every 5m; pending 0s; no data OK | Check C4 and S3 (DB blip?); page Ron to reconcile the user. | G:infra/grafana-provisioning/alerting/set-app-alerts.yaml:987-1040 |
| SET nonprod - O26 cross-client acceptance | InviteCrossClient. 3 or more invitations refused: the acceptor belongs to another client (1 hour). Evaluator: gt 2. | security | Every 5m; pending 0s; no data OK | Remove/deactivate the invitation if the user is unknown. | G:infra/grafana-provisioning/alerting/set-app-alerts.yaml:1041-1094 |
| SET nonprod - O27 uninvited sign-ins | UninvitedSignIn. 10 or more 'sign-in refused: no workspace' in 15 minutes. Evaluator: gt 9. | security | Every 5m; pending 0s; no data OK | None expected (new user or probing); WorkOS covers brute force. | G:infra/grafana-provisioning/alerting/set-app-alerts.yaml:1095-1148 |
| SET nonprod - O28 groups sync failing | GroupsSyncFailed. 3 or more workos group sync failures in 1 hour. Evaluator: gt 2. | infra | Every 5m; pending 0s; no data OK | RERUN groups:sync (429 is a known cause). | G:infra/grafana-provisioning/alerting/set-app-alerts.yaml:1149-1202 |
| SET nonprod - O29 Redis errors | RedisErrors. 3 or more redis client errors in 5 minutes. Evaluator: gt 2. | infra | Every 5m; pending 0s; no data OK | Check ElastiCache events; RESTART. | G:infra/grafana-provisioning/alerting/set-app-alerts.yaml:1203-1256 |
| SET nonprod - O34 rotation failed | RotationFailed, Errors. A DB password rotation failed or the rotator Lambda errored (credentials stop rotating). | infra | Every 5m; pending 0s; no data OK | RERUN rotation for that secret (README rotate-secret); if it fails again page Ron. | G:infra/grafana-provisioning/alerting/set-app-alerts.yaml:1257-1336 |
| SET nonprod - O30 API CPU/memory | CPUUtilization, MemoryUtilization. API service CPU or memory above 85 percent for 15 minutes. | infra | Every 5m; pending 0s; no data OK | Page Ron to resize (Terraform). Deploys cause brief spikes. | G:infra/grafana-provisioning/alerting/set-app-alerts.yaml:1337-1418 |
| SET nonprod - O32 DB burst credits / memory | CPUCreditBalance, FreeableMemory. t4g.small CPU credit balance under 50 or freeable memory under 100 MB. | infra | Every 5m; pending 0s; no data OK | Page Ron. A migration is a known cause. | G:infra/grafana-provisioning/alerting/set-app-alerts.yaml:1419-1498 |
| Watchdog | Always firing on purpose. Its hourly SNS publish is the heartbeat an AWS alarm watches (O33); Slack never receives it. Evaluator: gt 0. | heartbeat | Every 5m; pending 0s; no data Alerting | If the O33 alarm fires: open Grafana by SSM port-forward and restart the container. | G:infra/grafana-provisioning/alerting/set-app-alerts.yaml:1499-1534 |
| SET nonprod - O8b broker line missing | FileServeOk, AssumeWindowDistinctClients. At least 5 file reads succeeded in 2 hours with no assume_window line: the broker line (and assume_burst) may be dead. Needs default_value 0 on both metric filters. | infra | Every 30m; pending 0s; no data OK | Wrong image? RESTART or ROLLBACK. | G:infra/grafana-provisioning/alerting/set-app-alerts.yaml:1535-1614 |
| SET nonprod - O9 purge tripwire | PurgeTripped. The purge mass-deletion guard tripped. Evaluator: gt 0. | infra | Every 30m; pending 0s; no data OK | Run --dry-run, read the counts, RERUN with --approve <runId>:<count> only if reasonable; else page Ron. | G:infra/grafana-provisioning/alerting/set-app-alerts.yaml:1615-1668 |
| SET nonprod - O10 mover blocking jobs | MoverSkipped. At least 2 job runs skipped because a file move is running (48 hours). Evaluator: gt 1. | infra | Every 30m; pending 0s; no data OK | stop-task the filemove task if hung. Temporary: delete this rule at P2.23 step 6. | G:infra/grafana-provisioning/alerting/set-app-alerts.yaml:1669-1722 |
| SET nonprod - O22 role drift | RoleDrift. WorkOS role permissions differ from the committed mapping. Evaluator: gt 0. | security | Every 30m; pending 0s; no data OK | Fix in the WorkOS Dashboard (log field drift lists it); the 6-hour recheck clears it, or RESTART. | G:infra/grafana-provisioning/alerting/set-app-alerts.yaml:1723-1781 |
| SET nonprod - O12 certificate expiry | DaysToExpiry. ACM certificate under 21 days to expiry (auto-renew starts at 60 days: the validation CNAME is gone). Evaluator: lt 21. | infra | Every 1h; pending 0s; no data Alerting | Page Ron to restore the DNS validation record; calendar the Cloudflare AOP root CA expiry. | G:infra/grafana-provisioning/alerting/set-app-alerts.yaml:1782-1836 |
set-app-heartbeats.yaml
NOT VERIFIED applied state.
| Rule | Signal / threshold | Route label | Timing | Operator action | Definition |
|---|---|---|---|---|---|
| SET nonprod - O21 log pipeline alive | IncomingLogEvents. No log events in /set-nonprod/ecs for 15 minutes: every log-based alert is blind. Evaluator: lt 1. | critical | Every 5m; pending 0s; no data Alerting | RESTART; check the log group and KMS policy; page Ron. | G:infra/grafana-provisioning/alerting/set-app-heartbeats.yaml:11-71 |
| SET nonprod - O19 cleanup heartbeat | JobDone. No cleanup job_done outcome=ok in 26 hours (a skipped run never counts). Evaluator: lt 1. | infra | Every 30m; pending 0s; no data Alerting | RERUN cleanup. Skipped: see O10. | G:infra/grafana-provisioning/alerting/set-app-heartbeats.yaml:72-127 |
| SET nonprod - O20 purge heartbeat | JobDone. No purge job_done outcome=ok in 26 hours (a skipped run never counts). Evaluator: lt 1. | infra | Every 30m; pending 0s; no data Alerting | RERUN purge. Skipped: see O10. | G:infra/grafana-provisioning/alerting/set-app-heartbeats.yaml:128-188 |
| SET nonprod - O35 rotation overdue | RotationSucceeded. No successful DB password rotation in 100 days (90-day schedule). Evaluator: lt 1. | infra | Every 1h; pending 0s; no data Alerting | RERUN rotation (README), then ROTATE log. | G:infra/grafana-provisioning/alerting/set-app-heartbeats.yaml:189-244 |
set-nonprod-rotation.yaml
NOT VERIFIED applied state.
| Rule | Signal / threshold | Route label | Timing | Operator action | Definition |
|---|---|---|---|---|---|
| SET nonprod - DB password rotation outside the 90-day schedule | NonScheduledRotation. RotateSecret on {{ $labels.Secret }} by {{ $labels.Caller }}. Scheduled rotations never call it. Expected only for an incident rotation (README) or a new login's enrolment (set-nonprod-migrate-task). Anything else: treat as an incident. Evaluator: gt 0. | critical | Every 5m; pending 0s; no data OK | Details: CloudTrail (org trail) or the copy in /set-nonprod/ecs, account 661255175401, filter { $.detail.eventName = "RotateSecret" }. | G:infra/grafana-provisioning/alerting/set-nonprod-rotation.yaml:20-58 |
| SET nonprod - network interface removed with the rotation role | RotationRoleEniRemoval. {{ $labels.Call }} by {{ $labels.Caller }} with set-nonprod-db-rotation's credentials. Lambda's own ENI management and its DryRun checks are excluded, so this is the function's code or its stolen credentials. Treat as an incident. Evaluator: gt 0. | critical | Every 5m; pending 0s; no data OK | Details: CloudTrail (org trail) or the copy in /set-nonprod/ecs, account 661255175401, filter { $.source = "aws.ec2" }. | G:infra/grafana-provisioning/alerting/set-nonprod-rotation.yaml:59-98 |
| SET nonprod - rotation alert delivery failing | FailedInvocations. EventBridge rule {{ $labels.RuleName }} could not deliver to /set-nonprod/ecs, so the rotation alerts are blind. Check the log group resource policy (rotation.tf). Evaluator: gt 0. | critical | Every 5m; pending 0s; no data OK | No dedicated action recorded. Inspect the signal and escalate unexplained changes to Ron; do not alter code during response. | G:infra/grafana-provisioning/alerting/set-nonprod-rotation.yaml:99-132 |
Push detectors
NOT VERIFIED exact loaded-rule equivalence. The pack verifies the set-alerts family and sample paths, not every pattern or threshold. Source definitions generate success/refusal pairs for CloudTrail; service events generate one rule. E:evidence-aws.md:75-76 I:infra/modules/security-alerts/main.tf:270-317
| Generated rule name | Signal | Threshold | Channel suffix | Operator action | Definition |
|---|---|---|---|---|---|
| set-alerts-d1-ok, set-alerts-d1-refused | Root used or root session issued | One matching success/refusal event | security, critical | Expected: never, except a deliberate break-glass by Ron. Otherwise: incident - check the IP and what the session did. | I:infra/modules/security-alerts/main.tf:83-98 |
| set-alerts-d2-ok | Console sign-in without MFA | One matching success event | security | Expected: never. Find the user, force MFA, rotate its password. | I:infra/modules/security-alerts/main.tf:99-114 |
| set-alerts-d3-ok, set-alerts-d3-refused | IAM changed | One matching success/refusal event | security | Not OrganizationAccountAccessRole (Terraform), or refused: incident unless you know who did it. | I:infra/modules/security-alerts/main.tf:115-124 |
| set-alerts-d3-tf-ok | [terraform] IAM changed by OrganizationAccountAccessRole - check it matches an apply you ran | One matching success event | infra | Expected during a Terraform apply. No apply running: incident - someone holds the admin role. | I:infra/modules/security-alerts/main.tf:127-137 |
| set-alerts-d4-ok, set-alerts-d4-refused | Audit trail or organisation membership tampered | One matching success/refusal event | security, critical | Expected: only a reviewed org-security apply. Otherwise: incident - check the trail is still logging. | I:infra/modules/security-alerts/main.tf:138-154 |
| set-alerts-d5-ok, set-alerts-d5-refused | KMS key disabled, deleted or its policy changed | One matching success/refusal event | security | Expected: a Terraform apply you ran. Otherwise: incident - cancel any scheduled deletion first. | I:infra/modules/security-alerts/main.tf:155-163 |
| set-alerts-d6-ok, set-alerts-d6-refused | S3 exposure or RDS snapshot shared | One matching success/refusal event | security, critical | Expected: a Terraform apply you ran. Otherwise: incident - check the bucket policy / snapshot attributes. | I:infra/modules/security-alerts/main.tf:164-178 |
| set-alerts-d7-ok, set-alerts-d7-refused | Ingress opened to the internet | One matching success/refusal event | security | Expected: never (the ALB only admits Cloudflare). Revoke the rule, then find out who. | I:infra/modules/security-alerts/main.tf:179-195 |
| set-alerts-d9-ok, set-alerts-d9-refused | Data or its logs deleted | One matching success/refusal event | security | Expected: a Terraform apply you ran. Otherwise: incident - restore from snapshot/version. | I:infra/modules/security-alerts/main.tf:196-211 |
| set-alerts-d11-ok, set-alerts-d11-refused | Write outside eu-north-1 | One matching success/refusal event | security | Binary Networks staging runs only in eu-north-1: incident. | I:infra/modules/security-alerts/main.tf:212-233 |
| set-alerts-canary | Weekly canary - the alert path works, no action | One matching event | infra | Expected every Monday 07:00 UTC from each account. A call other than us-east-1 sns CreateTopic: check the schedule set-alerts-canary. | I:infra/modules/security-alerts/main.tf:236-248 |
| set-alerts-d14-ok, set-alerts-d14-refused | Alerting changed | One matching success/refusal event | critical | Expected: a PS.10 Terraform apply you ran. Otherwise: incident - the alerts may be blind; check CloudTrail. | I:infra/modules/security-alerts/main.tf:249-267 |
| set-alerts-d8a-ok, set-alerts-d8a-refused | Database login secret read by an unexpected identity | One matching success/refusal event | security | Expected readers are the API task, the migrate task and the rotator. Anyone else: incident - rotate the secret (README). | I:infra/nonprod/alerts.tf:39-47 |
| set-alerts-d8b-ok, set-alerts-d8b-refused | [INCIDENT] owner credential read | One matching success/refusal event | security | Only set-nonprod-migrate-exec reads it (once per migration). This was someone else, or a refused read: incident - rotate set/nonprod/db1/owner. | I:infra/nonprod/alerts.tf:48-56 |
| set-alerts-d8b-expected-ok | [expected] owner credential read for a migration | One matching success event | deploys | Expected once per migration (deploy) and once nightly at 03:00 UTC (the restore-point purge, purge.tf). Neither just now: incident. | I:infra/nonprod/alerts.tf:57-73 |
| set-alerts-d8c-ok, set-alerts-d8c-refused | BatchGetSecretValue called | One matching success/refusal event | security | The app never calls it. Incident: find which secrets the caller could read. | I:infra/nonprod/alerts.tf:74-80 |
| set-alerts-d8d-ok, set-alerts-d8d-refused | Database secret changed | One matching success/refusal event | security | Expected: a Terraform apply you ran (owner secret). Otherwise: incident. | I:infra/nonprod/alerts.tf:83-96 |
| set-alerts-d10-ok, set-alerts-d10-refused | Deploy path changed | One matching success/refusal event | security | Expected: a Terraform apply you ran. Anything else: incident - check which image the service runs. | I:infra/nonprod/alerts.tf:99-110 |
| set-alerts-d10-deploy-ok | [deploy] by pipeline - check it matches a develop run | One matching success event | deploys | Expected: one per merge to develop (Deploy staging run in set-app). No matching run: incident - stolen deploy credentials. | I:infra/nonprod/alerts.tf:111-125 |
| set-alerts-r1-ok, set-alerts-r1-refused | DB password rotation outside the 90-day schedule | One matching success/refusal event | security | Not the migrate task's enrolment (r1-enrol). Expected only for an incident rotation (README). Anything else: incident. | I:infra/nonprod/alerts.tf:128-135 |
| set-alerts-r1-enrol-ok | DB login enrolled for rotation by the migrate task | One matching success event | infra | Expected once per new login (a migration created it). No migration ran: incident. | I:infra/nonprod/alerts.tf:137-147 |
| set-alerts-r2-ok, set-alerts-r2-refused | Network interface removed with the rotation role | One matching success/refusal event | security | Lambda's own ENI management and DryRun checks are excluded: this is the function's code or its stolen credentials. Incident. | I:infra/nonprod/alerts.tf:148-157 |
| set-alerts-u3 | API task crashed | One matching event | infra | Several in a row = crash loop. Check the service events and /set-nonprod/ecs (stream prefix api). | I:infra/nonprod/alerts.tf:158-170 |
| set-alerts-u4 | API deployment failed or rolled back | One matching event | critical, deploys | The circuit breaker rolled back. Check the Deploy staging run and the service events. | I:infra/nonprod/alerts.tf:171-184 |
| set-alerts-u5 | Database migration failed | One matching event | critical | Read /set-nonprod/ecs stream prefix migrate. The release did not complete. | I:infra/nonprod/alerts.tf:185-202 |
| set-alerts-u6 | Cleanup job failed | One matching event | infra | A Standards Library purge did not finish; objects of items removed over 30 days ago stay until a run succeeds (the next one retries). Read /set-nonprod/ecs stream prefix cleanup: one line per task, the error class only. | I:infra/nonprod/alerts.tf:205-222 |
| set-alerts-u7 | Nightly purge failed (restore points, answer files) | One matching event | infra | Restore points older than a year, and stray or removed client answer files, stay until a run succeeds (the next night retries; expired points are already hidden from every read). A tripped tenant (too many files selected) also exits 1: read the counts, then use the reviewed --dry-run / --approve path. Read /set-nonprod/ecs stream prefix purge: one FAILED line per tenant. | I:infra/nonprod/alerts.tf:225-242 |
| set-alerts-u8 | Many client file sessions in five minutes | One matching event | infra | One API task assumed file-access roles for 30 or more distinct clients within five minutes (a loop, a bug, or a bulk read). The count comes from the process it watches. Check /set-nonprod/ecs for the assume_window lines and CloudTrail AssumeRole events by SourceIdentity; if it is not a known job, treat as an incident and revoke sessions on set-nonprod-answers-s3. | I:infra/nonprod/alerts.tf:246-261 |
| set-alerts-u9 | Standards extraction failed | One matching event | infra | That standard's file stays pending. Re-run the extract task for it (README Runbooks, RERUN); if it fails again, Remove the file (existing soft delete, restorable 30 days). Read /set-nonprod/ecs stream prefix extract. | I:infra/nonprod/alerts.tf:264-281 |
| set-alerts-s2p | Automated backups switched off on db1 | One matching event | infra | Expected: never. Re-enable the backup retention (data.tf) and find who changed it. | I:infra/nonprod/alerts.tf:282-291 |
| set-alerts-s3p | db1 storage nearly full, exhausted or autoscaling failed | One matching event | infra | Check FreeStorageSpace; raise max_allocated_storage (data.tf) if autoscaling hit its ceiling. | I:infra/nonprod/alerts.tf:292-300 |
| set-alerts-u10 | Grafana stopped reporting | One matching event | critical | No Watchdog publish for 3 hours: Grafana or its box is down, so every Grafana alert is blind (push alerts still work). Open Grafana over SSM port-forward and restart the container (nonprod README, Runbooks, GRAFANA). | I:infra/shared-services/alerts.tf:111-126 |
| set-alerts-d10e-ok, set-alerts-d10e-refused | Container registry changed | One matching success/refusal event | security | Expected: a Terraform apply you ran. Anything else: incident - check set-api's images and policy. | I:infra/shared-services/alerts.tf:129-142 |
| set-alerts-d12-ok, set-alerts-d12-refused | Terraform state touched by an unexpected identity | One matching success/refusal event | security | Data event: not in CloudTrail event history; search the eventID in /aws/cloudtrail/iontwrks-org-trail (management account). Only OrganizationAccountAccessRole (Terraform) touches it: incident. | I:infra/shared-services/alerts.tf:148-155 |
| set-alerts-d12-ops-ok | Terraform state touched beyond what Terraform does | One matching success event | security | Data event: search the eventID in /aws/cloudtrail/iontwrks-org-trail (management account). The Terraform role deleted state, touched another key or made another call: incident unless you did it. | I:infra/shared-services/alerts.tf:156-173 |
LIVE ON STAGING management alarms: iontwrks-assume_role_no_externalid, iontwrks-cloudtrail_tamper and iontwrks-root_usage; thresholds NOT VERIFIED. Inspect the matching trail event and confirm whether Ron authorized it; escalate unexpected security changes. E:evidence-aws.md:70
LIVE ON STAGING scanners push rules: scanners-aws-health, scanners-ecs-deployment, scanners-ecs-task-stopped, scanners-guardduty-findings. Exact patterns, thresholds, targets and channel mappings are NOT VERIFIED; the pack supplies names only. Obtain rule/target exports and a sanitized delivery test. E:evidence-aws.md:76,81
MERGED, NOT DEPLOYED GitHub workflow notifications use the named repository secret SLACK_WEBHOOK_URL (name only). Failed/timed-out named workflows trigger a message; weekday 07:00 UTC digest runs from the default branch. Channel and actual delivery are NOT VERIFIED. A:.github/workflows/alerts.yml:3-29,36-49 The Security reusable workflow also receives that secret by name. A:.github/workflows/security.yml:1-10
Heartbeats, health checks and conflicts
APPROVED, IN BUILD — independent Grafana dead-man: Watchdog to SNS, then AWS NumberOfMessagesPublished below 1 for three hourly periods with missing data treated as breaching, routed through Amazon Q. Topic and alarm were not observed live. I:infra/shared-services/heartbeat.tf:9-50 E:evidence-aws.md:82
NOT VERIFIED — Cloudflare U1 and O5 configuration/delivery. O5 proposes readiness, but the deployment docs describe public readiness blocking. Verify an authenticated probe path before claiming it works. A:.set/work/PS.14.md:60 I:infra/nonprod/edge.tf:170-175,247-256
Source disagreement: PS.14's initial burst rows specify one-minute evaluation, whereas Ron selected five minutes for staging and the new rule group uses five minutes. This page records all three; none proves the loaded interval. A:.set/work/PS.14.md:48,62-65 A:.set/work/decisions-2026-10-26.md:68-70 G:infra/grafana-provisioning/alerting/set-app-alerts.yaml:1-14
The older C4 connection limit is explicitly estimated (164 as 80% of about 205). It is NOT VERIFIED as a measured database ceiling; verify max_connections and review after instance resizing. G:infra/grafana-provisioning/alerting/set-alerts.yaml:10-11
Later app-pack provisioning evidence
LIVE ON STAGING for the observed bootstrap/history only: live instance user-data writes only datasources/loki.yaml and pins Grafana 11.1.0. SSM history records copying contact points, policies, set-alerts.yaml, set-nonprod-rotation.yaml and CloudWatch datasource files with restarts on October 22/24. That differs from the newer Terraform bootstrap described above. Terraform ignores user_data changes. NOT VERIFIED current loaded-rule state: a successful copy/restart is not a rule export. E:evidence.md:76-78
APPROVED, IN BUILD — PS.14 provisioning at fb394cd was never copied in the observed history. The new Watchdog, dashboard and feature rules are therefore not established live. E:evidence.md:77
LIVE ON STAGING push target mappings: u3 → infra; u4 → critical and deploys; u5 → critical; u6/u7/canary → infra. These map to the shared hub channels listed above. The current Terraform u7 title includes answer files, but the live purge task still runs only purge-restore-points.js without FILES_BUCKET. E:evidence.md:29,79
NOT VERIFIED Cloudflare routing, health checks, Grafana current rule export and Slack contact-point names. Verify through sanitized read-only exports and delivery observations. E:evidence.md:45,78