Cost Control — Park the Lab When Idle

golemsec is a lab. The biggest cost lever is simply stopping compute when you're not using it. EC2 charges for compute only while instances run; stopped instances keep their disks (EBS), private IPs, EIP, and all configuration — nothing rebuilds.

The numbers

State	Monthly cost (approx)
Running 24/7	~$630–780 compute + $99 storage + pfSense Plus software fee ≈ $730–880/mo
Stopped (parked)	~$99/mo EBS storage only (1,240 GB) + ~$3.60 idle Elastic IP

Parking the fleet is an ~85% cost reduction — and reversible in minutes.

The switch

D:\golemsec\golemsec-power.ps1 (uses the binary AWS profile, targets everything tagged Project=golemsec):

.\golemsec-power.ps1 status   # list fleet + state + storage cost (read-only)
.\golemsec-power.ps1 stop      # stop all compute — pay only storage
.\golemsec-power.ps1 start     # ordered start

What "stop" does

Stops all running golemsec instances in one call. Compute billing (and the pfSense Plus software fee) stops once they reach stopped. EBS storage continues to bill.

What "start" does (ordered, on purpose)

Phase 1 — foundation: pfSense (routing/NAT/egress) + DC01 + DC02 (AD + DNS). Waits ~3 minutes.
Phase 2 — everything else: all app/data/mgmt/security/endpoint instances.

This order matters because the other hosts depend on DNS/AD (domain-joined Windows, app lookups) and pfSense egress (internet + SSM management). Docker containers (restart: unless-stopped) and Windows/Exchange/SQL services auto-start on boot, so apps come back on their own. Exchange/Windows may take several minutes to be fully ready.

What survives a stop/start

Private IPs (10.50.x.x) — retained, so all AD/DNS/app config stays valid.
Elastic IP (35.170.40.167) on pfSense — retained.
All disks/data, installed apps, AD, mailboxes, the SIEM, the PII DB — untouched.

Scheduled auto-off (ACTIVE)

A serverless nightly auto-stop is deployed so the lab can't be left billing overnight:

Resource	Value
EventBridge rule	`golemsec-nightly-off` — `cron(0 21 * * ? )` (21:00 UTC ≈ midnight Israel*), ENABLED
Lambda	`golemsec-power` (Python) — stops/starts all `Project=golemsec` instances by tag
IAM role	`golemsec-power-lambda` (`ec2:Describe/Stop/StartInstances`)
Setup script	`D:\golemsec\golemsec-schedule-setup.ps1` (re-runnable; edit `$CRON` to change the time)

It fires every night at midnight Israel and stops everything — even if in use, so if you work past midnight just run start again.
Bring it back: .\golemsec-power.ps1 start (ordered) or aws lambda invoke --function-name golemsec-power --payload file://payload.json --cli-binary-format raw-in-base64-out out.json with {"action":"start"}.
Pause the schedule: aws events disable-rule --name golemsec-nightly-off · resume: enable-rule.
(No auto-start is scheduled by design — you start it manually when you need it, so it never bills on a day you're not using it.)

Going further (optional)

Auto-start weekday mornings — add a second EventBridge rule with {action:start} if you want it ready by work hours (not enabled by default to avoid billing on unused days).
Lower the storage floor — the ~$99/mo is EBS for 1,240 GB. Right-sizing oversized volumes or removing unused boxes (e.g., spare endpoints) reduces the parked cost.
Don't terminate to save money unless you intend to rebuild — terminating deletes the instances; stop/start is the keep-everything option.

Scripts

Download:

golemsec-power.ps1 — the on/off/status switch
golemsec-schedule-setup.ps1 — one-time nightly auto-off setup

No secrets in these — they use the local binary AWS profile and target the Project=golemsec tag. Adjust the profile/region/cron at the top to fit your environment.

`golemsec-power.ps1`

<#
  golemsec-power.ps1 — ON/OFF switch for the whole golemsec lab (cost control).
  Usage:
    .\golemsec-power.ps1 status   # show fleet + cost estimate (read-only)
    .\golemsec-power.ps1 stop      # stop all compute (pay only storage)
    .\golemsec-power.ps1 start     # ordered start: pfSense + DCs first, then the rest
#>
param([ValidateSet("status","stop","start")][string]$Action="status")
$env:AWS_PROFILE   = "binary"
$env:AWS_DEFAULT_REGION = "us-east-1"
$FILTER = "Name=tag:Project,Values=golemsec"

function Get-Fleet {
  aws ec2 describe-instances --filters $FILTER `
    --query "Reservations[].Instances[].{Id:InstanceId,Name:Tags[?Key=='Name']|[0].Value,State:State.Name,Type:InstanceType}" `
    --output json | ConvertFrom-Json | Where-Object { $_.State -ne "terminated" }
}

$fleet = Get-Fleet
if (-not $fleet) { Write-Host "No golemsec instances found." -ForegroundColor Yellow; return }

switch ($Action) {
  "status" {
    $fleet | Sort-Object Name | Format-Table Name, Id, Type, State -AutoSize
    $running = @($fleet | Where-Object State -eq "running").Count
    $stopped = @($fleet | Where-Object State -eq "stopped").Count
    Write-Host ("Running: {0}  |  Stopped: {1}  |  Total: {2}" -f $running,$stopped,$fleet.Count) -ForegroundColor Cyan
    $vids = aws ec2 describe-instances --filters $FILTER --query "Reservations[].Instances[].BlockDeviceMappings[].Ebs.VolumeId" --output text
    if ($vids) {
      $gb = (aws ec2 describe-volumes --volume-ids ($vids -split "\s+") --query "sum(Volumes[].Size)" --output text)
      Write-Host ("EBS total: {0} GB  ->  storage-only cost ~`${1:N0}/mo when fully stopped" -f $gb, ([double]$gb*0.08)) -ForegroundColor Cyan
    }
    Write-Host "Compute when running 24/7 ~`$630-780/mo. Run '.\golemsec-power.ps1 stop' to park it." -ForegroundColor DarkGray
  }
  "stop" {
    $ids = @($fleet | Where-Object State -in "running","pending").Id
    if (-not $ids) { Write-Host "Nothing running."; return }
    Write-Host ("Stopping {0} instances..." -f $ids.Count) -ForegroundColor Yellow
    aws ec2 stop-instances --instance-ids $ids --query "StoppingInstances[].InstanceId" --output text
    Write-Host "Done. Compute billing stops once they reach 'stopped'. Storage still bills." -ForegroundColor Green
  }
  "start" {
    $found = @($fleet | Where-Object { $_.Name -match "pfsense|dc01|dc02" }).Id
    $rest  = @($fleet | Where-Object { $_.Name -notmatch "pfsense|dc01|dc02" -and $_.State -in "stopped","stopping" }).Id
    if ($found) {
      Write-Host "Phase 1: starting foundation (pfSense + DC01 + DC02 = routing, AD, DNS)..." -ForegroundColor Yellow
      aws ec2 start-instances --instance-ids $found --query "StartingInstances[].InstanceId" --output text
      Write-Host "Waiting 180s for AD/DNS/egress to come up before the rest..." -ForegroundColor DarkGray
      Start-Sleep -Seconds 180
    }
    if ($rest) {
      Write-Host ("Phase 2: starting remaining {0} instances..." -f $rest.Count) -ForegroundColor Yellow
      aws ec2 start-instances --instance-ids $rest --query "StartingInstances[].InstanceId" --output text
    }
    Write-Host "Started. Windows/Exchange may take several minutes; Docker apps auto-start on boot." -ForegroundColor Green
  }
}

`golemsec-schedule-setup.ps1`

<#
  One-time setup: nightly auto-OFF for the golemsec fleet.
  Creates: IAM role -> Lambda (stop/start by tag) -> EventBridge schedule (nightly stop).
  Re-runnable (idempotent). Change $CRON below to adjust the time (UTC).
#>
$env:AWS_PROFILE = "binary"; $env:AWS_DEFAULT_REGION = "us-east-1"
$ROLE = "golemsec-power-lambda"
$FN   = "golemsec-power"
$RULE = "golemsec-nightly-off"
$CRON = "cron(0 21 * * ? *)"   # 21:00 UTC = ~00:00 Israel (IDT). Edit to taste.
$tmp  = Join-Path $env:TEMP "gsched"; New-Item -ItemType Directory -Force -Path $tmp | Out-Null

# --- 1) IAM role ---
'{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"Service":"lambda.amazonaws.com"},"Action":"sts:AssumeRole"}]}' | Out-File "$tmp\trust.json" -Encoding ascii
$roleArn = aws iam get-role --role-name $ROLE --query "Role.Arn" --output text 2>$null
if ($LASTEXITCODE -ne 0 -or -not $roleArn) {
  $roleArn = aws iam create-role --role-name $ROLE --assume-role-policy-document file://$tmp/trust.json --tags Key=Project,Value=golemsec --query "Role.Arn" --output text
  aws iam attach-role-policy --role-name $ROLE --policy-arn arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole | Out-Null
  '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":["ec2:DescribeInstances","ec2:StopInstances","ec2:StartInstances"],"Resource":"*"}]}' | Out-File "$tmp\perms.json" -Encoding ascii
  aws iam put-role-policy --role-name $ROLE --policy-name ec2-power --policy-document file://$tmp/perms.json
  Write-Host "IAM role created; waiting 15s for propagation..."; Start-Sleep 15
}

# --- 2) Lambda (stop/start golemsec by tag) ---
@'
import boto3
def lambda_handler(event, context):
    action = (event or {}).get("action", "stop")
    ec2 = boto3.client("ec2")
    r = ec2.describe_instances(Filters=[
        {"Name": "tag:Project", "Values": ["golemsec"]},
        {"Name": "instance-state-name", "Values": ["running", "stopped"]}])
    ids = [i["InstanceId"] for res in r["Reservations"] for i in res["Instances"]]
    if not ids:
        return {"action": action, "count": 0}
    if action == "start":
        ec2.start_instances(InstanceIds=ids)
    else:
        ec2.stop_instances(InstanceIds=ids)
    return {"action": action, "count": len(ids), "ids": ids}
'@ | Out-File "$tmp\lambda_function.py" -Encoding ascii
Compress-Archive -Path "$tmp\lambda_function.py" -DestinationPath "$tmp\func.zip" -Force
aws lambda get-function --function-name $FN --query "Configuration.FunctionName" --output text 2>$null | Out-Null
if ($LASTEXITCODE -eq 0) {
  aws lambda update-function-code --function-name $FN --zip-file fileb://$tmp/func.zip | Out-Null
} else {
  for ($i=0; $i -lt 6; $i++) {
    aws lambda create-function --function-name $FN --runtime python3.12 --role $roleArn --handler lambda_function.lambda_handler --zip-file fileb://$tmp/func.zip --timeout 120 --tags Project=golemsec 2>$null | Out-Null
    if ($LASTEXITCODE -eq 0) { break }
    Start-Sleep 8
  }
}
$fnArn = aws lambda get-function --function-name $FN --query "Configuration.FunctionArn" --output text

# --- 3) EventBridge schedule -> nightly stop ---
aws events put-rule --name $RULE --schedule-expression $CRON --state ENABLED --description "Park golemsec fleet nightly" | Out-Null
$ruleArn = aws events describe-rule --name $RULE --query "Arn" --output text
aws lambda add-permission --function-name $FN --statement-id evt-nightly-off --action lambda:InvokeFunction --principal events.amazonaws.com --source-arn $ruleArn 2>$null | Out-Null
$targetsJson = '[{"Id":"1","Arn":"' + $fnArn + '","Input":"{\"action\":\"stop\"}"}]'
$targetsJson | Out-File "$tmp\targets.json" -Encoding ascii
aws events put-targets --rule $RULE --targets file://$tmp/targets.json | Out-Null
Write-Host "DONE. Nightly auto-off active."

The numbers​

The switch​

What "stop" does​

What "start" does (ordered, on purpose)​

What survives a stop/start​

Scheduled auto-off (ACTIVE)​

Going further (optional)​

Scripts​

golemsec-power.ps1​

golemsec-schedule-setup.ps1​