Monitoring and Alerting

The Scanners platform runs a four-layer alerting system. Each layer watches a different part of the stack and routes to a dedicated Slack channel in the BinaryNetworks workspace. All four layers are live and Slack-verified.

This page describes what is in production today. See also Production Deployment for the topology and Logging & SIEM for the structured-log stream.

Slack channels

Channel	Purpose	Fed by
#alerts-critical	Site/edge down, page-worthy	Layer 4 (Cloudflare uptime)
#alerts-infra	Infra health (ECS/RDS/Redis)	Layer 2 (CloudWatch → Chatbot)
#alerts-prod	Application behaviour (errors)	Layer 1 (Grafana)
#alerts-security	Security events	Layer 1 (Grafana)
#deploys	CI/CD deploy notifications	GitHub Actions

Layer 1 — Application alerting (Grafana + Loki)

What it watches: application behaviour — ERROR-level app logs and security events emitted by the API and workers.

How logs get there: the ECS FireLens (Fluent Bit) sidecar on every task ships each app container's logs to Loki running on a private Grafana box.

:::note FireLens Option B (not S3 config) Fargate does not support config-file-type = "s3", so we do not store a Fluent Bit config in S3. Instead each task definition ships to Loki via per-container output options (local.firelens_loki_options in ecs.tf, host = the Grafana box's private IP on port 3100). The earlier S3-config approach was removed. :::

The Grafana box is a private observability host:

• EC2 t4g.small, Amazon Linux 2023 (arm64), in a private subnet.
• No public IP, no Elastic IP, no public DNS record — it is invisible to external attack-surface scanning by design.
• Loki stores chunks + index in a dedicated S3 bucket with a 30-day lifecycle.
• Reached only via AWS SSM Session Manager port-forwarding (see below).
• Terraform ignores user_data / ami changes so a plan never stops or replaces the live box; config drift is fixed in place with SSM Run Command.

Alerting is provisioned from infra/grafana-provisioning/:

• A Loki datasource (uid: loki).
• Contact points slack-prod (→ #alerts-prod) and slack-security (→ #alerts-security). The Slack bot token is substituted on the box at boot from Secrets Manager; the repo keeps a __SLACK_BOT_TOKEN__ placeholder — no token in git.
• Team-based routing: team=prod → slack-prod, team=security → slack-security.
• 11 alert rules querying the structured action / outcome / level / service Loki labels:
  • Catch-all + security (→ #alerts-prod / #alerts-security): Application Errors (any ERROR log in 5m), Security Event (Key Rotated) (org.key_rotated in 10m), API Denial Spike (outcome="denied" probing/bad creds).
  • Degradation / abuse (→ #alerts-prod): Scan Tool Failures (tool.failed), Enrichment API Degradation (enrich.api_error), API Rate-Limit Spike (ratelimit.exceeded), Zero-Finding Scan Spike (scan.zero_findings), Scan Failure Spike (scan.failed).
  • Dead-man's-switch (→ #alerts-critical): API / EASM Worker / DAST Worker Silent — each service emits a 60s action="heartbeat"; ~5m of silence means hung/OOM-killed, not idle.

:::note noData semantics — and the DatasourceNoData false-alarm fix Grafana has two distinct "no result" states: DatasourceNoData means the query ran and returned an empty set — Loki was reached, there were just no matching lines (governed by noDataState); DatasourceError means the datasource was unreachable (governed by execErrState) — that is the real "Loki is down" signal. Pre-launch, an empty count is the normal state, so (fixed 2026-06-12) the 8 count-based spike rules use noDataState: OK and only the 3 dead-man's-switch rules use noDataState: Alerting. Before the fix they used noDataState: NoData and fired spurious DatasourceNoData alerts on quiet streams. :::

Reaching the Grafana dashboard (SSM port-forward)

The box has no inbound network path. Open a tunnel from your laptop:

aws ssm start-session \
  --target <grafana-instance-id> \
  --document-name AWS-StartPortForwardingSession \
  --parameters '{"portNumber":["3000"],"localPortNumber":["3000"]}' \
  --region eu-north-1 --profile scanners-prod

Then open http://localhost:3000. The admin password is in Secrets Manager. For ad-hoc Loki queries use the same command with portNumber 3100. Close the session with Ctrl-C in the terminal running start-session.

Investigating Layer 1 alerts ("Jarvis")

The read-only AI investigator openclaw-ops ("Jarvis") can probe Loki directly to tell a real pipeline outage from a merely quiet stream: its check_loki tool issues read-only GETs to Loki on the box (/ready, /labels, count_over_time for the alert's stream + heartbeats). The Grafana box security group allows port 3100 ingress from the openclaw-ops task SG for exactly this. Jarvis is on-demand — you @-mention it in an alert thread and it investigates; it does not auto-investigate every alert. See openclaw-ops (Jarvis).

Layer 2 — Infrastructure alerting (CloudWatch → AWS Chatbot → Slack)

What it watches: infrastructure health. CloudWatch metric alarms (ECS task-down via EventBridge, RDS CPU / free-storage / connections, Redis CPU / memory) publish to the SNS topic scanners-alerts, which an AWS Chatbot (Amazon Q Developer chat) integration delivers to #alerts-infra.

note

AWS Chatbot's control plane lives in us-east-2; the account SCP exempts chatbot:* and q:* so the integration can be configured from the region-locked account.

Layer 3 — AWS Health alerting

What it watches: AWS-side service events and scheduled changes affecting our account/region. AWS Health events are captured by EventBridge and routed to SNS → Slack alongside Layer 2.

Layer 4 — External uptime (Cloudflare)

What it watches: is the public endpoint actually up, from outside our network. A Cloudflare Health Check on modules.binarynetworks.ai fires a notification webhook to #alerts-critical when the endpoint goes unhealthy. This is the page-worthy, customer-facing signal and is independent of AWS.

CloudWatch alarms (Layer 2 detail)

11 metric alarms back Layer 2: ECS service task-down (EventBridge), RDS CPU / free storage / connections (plus an RDS event subscription), and Redis CPU / memory. All publish to the scanners-alerts SNS topic.

Verifying end to end

Layer 1 was verified by pushing a synthetic ERROR log into Loki and confirming the Scanners - Application Errors rule entered firing, that the Grafana Slack bot passes auth.test, and that a message is delivered to #alerts-prod. Layers 2–4 were each verified by triggering their source condition (or a test notification) and confirming Slack delivery.