Disaster Recovery — Plan & Recovery Runbook
How to recover golemsec if something breaks — a single host, the data, or the whole environment.
Strategy: two recovery layers
| Layer | What it is | Use when | Speed |
|---|---|---|---|
| 1 — DR AMIs (fast restore) | Point-in-time machine images of all 21 instances (OS + data: AD, mailboxes, PII DB, app state), tagged DR=true | A host died, was corrupted, or terminated | Minutes |
| 2 — Infrastructure-as-Code (rebuild) | Terraform (D:\golemsec\terraform) + provisioning scripts (D:\golemsec\provisioning) + this documentation | Total loss (account/region), or you want a clean rebuild | Hours |
Layer 1 restores exact state. Layer 2 rebuilds the infrastructure from code but not the data (AD objects, mailboxes, the
golemsec_corpPII DB, app records) — that lives only in the AMIs. So keep the DR AMIs current.
Objectives
- RPO (data-loss window): = age of the latest DR AMI set. Current set: 2026-06-08. Refresh before/after major changes.
- RTO (time to recover): single host ~5–15 min; whole environment ~30–60 min (ordered restore + boot).
Recovery reference — host → placement → DR AMI
To restore a host you relaunch it in the same subnet + security group, with the same private IP, from its DR AMI (so AD/DNS and all app configs that reference 10.50.x.x stay valid).
| Host | Private IP | Subnet (tier) | Security Group | Type | DR AMI |
|---|---|---|---|---|---|
| dc01 | 10.50.20.10 | subnet-06117adf33484554f (server) | sg-0d4e58b5e79103183 | t3.small | ami-0d7a236d1406585b1 |
| dc02 | 10.50.20.11 | subnet-06117adf33484554f (server) | sg-0d4e58b5e79103183 | t3.small | ami-0211ef25dc737c7a7 |
| exchange | 10.50.20.128 | subnet-06117adf33484554f (server) | sg-0d4e58b5e79103183 | t3.large | ami-09a521fd4ba67007f |
| keycloak | 10.50.20.133 | subnet-06117adf33484554f (server) | sg-0d4e58b5e79103183 | t3.small | ami-0c802ffee6e5aaf76 |
| app-lamp | 10.50.20.38 | subnet-06117adf33484554f (server) | sg-0d4e58b5e79103183 | t3.medium | ami-08c14d446ed21660f |
| erpnext | 10.50.20.164 | subnet-06117adf33484554f (server) | sg-0d4e58b5e79103183 | t3.medium | ami-078a7e849754ee8ab |
| dev-server | 10.50.20.107 | subnet-06117adf33484554f (server) | sg-0d4e58b5e79103183 | t3.large | ami-043cba1664f37e9c7 |
| db-pgmy | 10.50.30.12 | subnet-0ff5e25539e41b310 (data) | sg-01263b76858fc0006 | t3.medium | ami-00626f54d9dfa361f |
| db-mssql | 10.50.30.38 | subnet-0ff5e25539e41b310 (data) | sg-01263b76858fc0006 | t3.medium | ami-0d0dfcda161beaea4 |
| nextcloud | 10.50.10.51 | subnet-0176a62ffcded08b7 (dmz) | sg-0606f4f745cf716e8 | t3.small | ami-089958bb14a3a37f6 |
| wazuh | 10.50.40.63 | subnet-0a9101d272a217634 (security) | sg-08813dfe0f55430e3 | t3.large | ami-059878500745425b4 |
| zabbix | 10.50.50.213 | subnet-06787e1482d1306d2 (mgmt) | sg-0cae08e03887bf98e | t3.small | ami-09582455ea42a106f |
| mgmt-win | 10.50.50.95 | subnet-06787e1482d1306d2 (mgmt) | sg-0cae08e03887bf98e | t3.medium | ami-0f01d5aa9e47011e7 |
| veeam | 10.50.50.49 | subnet-06787e1482d1306d2 (mgmt) | sg-0cae08e03887bf98e | t3.large | ami-011836702ce852ccc |
| endpoint-pc-01..06 | 10.50.60.x | subnet-02d28adc4e31fcbbb (endpoint) | sg-087a1717214a536dd | t3.small | golemsec-dr-…-endpoint-pc-0N-… |
| pfsense (special) | 10.50.0.119 (+EIP) | subnet-0cdf7beceaa68e16e (public) | sg-014f555a3a8b2dfc1 | t3.small | ami-04066f25850b993c9 |
Full AMI list any time:
aws ec2 describe-images --owners self --filters Name=tag:DR,Values=true --query "Images[].[Name,ImageId]" --output text
Procedure A — restore a single host (most common)
- Terminate the broken instance (if it still exists) so its private IP frees up:
aws ec2 terminate-instances --instance-ids <bad-id>(wait until terminated). - Relaunch from its DR AMI into the same subnet/SG/IP (example =
keycloak):$env:AWS_PROFILE='binary'; $env:AWS_DEFAULT_REGION='us-east-1'aws ec2 run-instances `--image-id ami-0c802ffee6e5aaf76 `--instance-type t3.small `--subnet-id subnet-06117adf33484554f `--private-ip-address 10.50.20.133 `--security-group-ids sg-0d4e58b5e79103183 `--iam-instance-profile Name=golemsec-ssm-role `--no-associate-public-ip-address `--tag-specifications "ResourceType=instance,Tags=[{Key=Name,Value=golemsec-keycloak},{Key=Project,Value=golemsec}]" - Boot + verify — Docker apps and Windows/AD services auto-start. Give Windows/Exchange a few minutes.
Use the row from the reference table for the host you're restoring (AMI / subnet / IP / SG / type). Keep
Tags=Project=golemsecso the power switch and nightly schedule keep managing it.
Procedure B — restore the whole environment
Restore in the same dependency order the power switch uses:
- Foundation first:
pfsense,dc01,dc02(routing/AD/DNS). Wait ~3–5 min. - Then the rest (data → server/app → mgmt → security → endpoints).
- Re-point routing to the new pfSense LAN ENI (see pfSense note below) — required because the private route tables default-route to pfSense's LAN interface, and a rebuilt pfSense has a new ENI.
(If instances were only stopped, you don't restore at all — just .\golemsec-power.ps1 start.)
pfSense is a special case
pfSense is the edge router (2 ENIs: WAN in public, LAN in server; the EIP; source/dest check disabled; route tables point at its LAN ENI). Restoring it from AMI also requires:
- disable source/dest check on the new instance,
- re-create/attach the LAN ENI (or note the new primary ENI id),
- re-associate the EIP
35.170.40.167, - update the private route tables so
0.0.0.0/0points to the new pfSense LAN ENI.
For a total rebuild, it's cleaner to re-apply Terraform (which recreates pfSense networking + routes correctly) and then restore the pfSense config, rather than hand-wiring ENIs.
Procedure C — data-only recovery
If just a database/app is corrupted (host is fine), restore at the data layer instead of the whole host:
- Launch the data-tier DR AMI (
db-pgmy/db-mssql) as a temporary instance, mount/extract the needed DB, and restore into the live host — or detach the live volume and replace it with one created from the DR AMI's snapshot.
Keeping DR current
- Refresh (best while parked, for a clean snapshot):
.\golemsec-power.ps1 stop # park for consistency# re-run the create-image loop (see golemsec-power.ps1 / scripts page) — produces a new golemsec-dr-*-<date> set
- Prune (stop paying for old images): deregister old AMIs and delete their snapshots —
Snapshot storage (~$0.05/GB-mo of used data) is the only ongoing DR cost.# list: aws ec2 describe-images --owners self --filters Name=tag:DR,Values=true# delete: aws ec2 deregister-image --image-id <ami>; then delete its snapshots
Testing the plan
- Non-destructive test: launch one DR AMI (e.g.,
nextcloud) into a throwaway private IP, confirm it boots and the app responds, then terminate it. Proves the image is good without touching production. - Re-validate after any major change to the environment.
Related
- Cost Control — the on/off switch + nightly auto-off (
golemsec-power.ps1). - AWS Architecture — VPC/subnets/SGs and the locked-down edge.
- Credentials & Break-Glass — admin logins needed after recovery.