Releasing & Deploying a new build

How a code change reaches the running services on AWS. The pipeline is two halves — an automatic build and a deploy — and today the deploy half is gated, so it's often run by hand. This page is the runbook for both.

The pipeline

• Build (automatic): every push to main runs the Lint + pytest and Build + push images jobs (.github/workflows/deploy.yml). Each image is pushed to ECR tagged sha-<commit> and latest.
• Deploy (gated): the Deploy to production job sits behind the GitHub production environment's required-reviewer protection, so on a normal push it is skipped. Result: the images exist in ECR but the running services are unchanged until someone deploys.

:::warning Why deploys are manual today Because the deploy job is gated, push-to-main builds but does not deploy, and each service's task definition stays pinned to the last-deployed image sha. To make merges deploy automatically, ungate the job (remove the required-reviewer rule on the production environment) or approve the deploy job after each merge. :::

What ECS actually runs

An ECS service runs the image written into its task definition — not "the latest". So deploying = getting the new image into the task def. Two cases:

Task-def image	How to deploy
pinned to :latest (e.g. cloud-worker)	force-new-deployment — a fresh task pulls latest
pinned to a sha-… (e.g. api, easm-worker, dast-worker)	register a new revision with the new image, then update the service

Check what a service is pinned to:

export AWS_PROFILE=scanners-prod
TD=$(aws ecs describe-services --cluster scanners --services api \
  --query 'services[0].taskDefinition' --output text)
aws ecs describe-task-definition --task-definition "$TD" \
  --query "taskDefinition.containerDefinitions[?name=='api'].image | [0]" --output text

Case A — task def is on :latest (one command)

aws ecs update-service --cluster scanners --service cloud-worker --force-new-deployment

Fargate pulls the image per task launch, so the next task gets the new build. For scale-to-zero workers (easm-worker, dast-worker, cloud-worker) you don't even wait — the next scan cold-starts a task on the new image.

Case B — task def is pinned to a sha (register a new revision)

This is exactly what the Deploy to production job does (deploy.yml); run it by hand per service. TAG is the new image tag (the commit sha, e.g. sha-61e8b57…):

export AWS_PROFILE=scanners-prod
REG=429134227608.dkr.ecr.eu-north-1.amazonaws.com
TAG=sha-<new-commit>

deploy_one () {   # $1 service  $2 ecr-repo  $3 container-name
  aws ecs describe-task-definition --task-definition "scanners-$1" \
    --query taskDefinition --output json > td.json
  python3 -c "
import json; td=json.load(open('td.json'))
for c in td['containerDefinitions']:
    if c['name']=='$3': c['image']='$REG/$2:$TAG'
for k in ['taskDefinitionArn','revision','status','requiresAttributes',
          'compatibilities','registeredAt','registeredBy']: td.pop(k,None)
json.dump(td, open('ntd.json','w'))"
  ARN=$(aws ecs register-task-definition --cli-input-json file://ntd.json \
        --query taskDefinition.taskDefinitionArn --output text)
  aws ecs update-service --cluster scanners --service "$1" \
        --task-definition "$ARN" --force-new-deployment
}

deploy_one api          scanners-api    api
deploy_one easm-worker  scanners-worker easm-worker
deploy_one dast-worker  scanners-worker dast-worker

The five steps are: describe → swap image → register new revision → update-service → (wait). The api/easm/dast workers share the scanners-worker image; the cloud worker uses scanners-cloud-worker.

Wait for the always-on services

Workers are scale-to-zero (no wait needed). The api and cloudflared are always-on — wait for the rolling deploy to finish:

aws ecs wait services-stable --cluster scanners --services api

Confirm the running task is on the new image:

TASK=$(aws ecs list-tasks --cluster scanners --service-name api --query 'taskArns[0]' --output text)
DIG=$(aws ecs describe-tasks --cluster scanners --tasks "$TASK" \
  --query 'tasks[0].containers[?name==`api`].imageDigest|[0]' --output text)
aws ecs describe-images --repository-name scanners-api --image-ids imageDigest=$DIG \
  --query 'imageDetails[0].imageTags' --output json

Notes

• The sweeper runs in the api; the scan engine + heartbeat run in the workers. A fix to either only goes live once that service is redeployed.
• Secrets injected at task start (e.g. CELERY_BROKER_URL) update on the next task launch, so changing a secret value also needs a force-new-deployment of the always-on services to take effect.
• The recommended end state is to ungate the deploy job so main → live, and switch the api/worker task defs to a managed-by-CI deploy (sha-pinned per release for clean rollback).