Six months ago I wrote that managed runners were the bigger saving in GitHub Actions over Jenkins, and for a new repository I still think that. This is the other half of that sentence. We outgrew hosted runners, moved everything onto our own machines, and the thing that made it bearable was that Terraform owned the machines rather than a person.

Three pressures pushed us off the hosted fleet, and none of them was ideology. Builds needed to reach a database that has no public address. Some jobs wanted more memory than the standard hosted runner offers. And the minutes had stopped being background noise on the invoice.

What replaced it is the terraform-aws-github-runner module: a set of Lambda functions that watch GitHub for queued jobs, start EC2 spot instances to run them, and terminate the instances when the work is done. The whole of our CI and CD sits on it and the bill lands between $150 and $200 a month.

The shape of it

Nothing in this module runs continuously except the Lambdas, and Lambdas that are not invoked cost nothing.

A workflow job is queued. GitHub sends a workflow_job event to a webhook, which is an API Gateway endpoint in front of a Lambda. That Lambda verifies the signature, decides whether the event is one it cares about, and posts it to an SQS queue. The queue holds the message for a configurable delay — thirty seconds by default — so that a runner already running has a chance to pick the job up before anything new is started.

The scale-up Lambda reads the queue. It checks whether the job has already been claimed and whether the deployment is at its ceiling, requests a runner registration token from GitHub, writes that token into SSM Parameter Store, and launches a spot instance from a launch template. The instance's user data pulls the token out of Parameter Store, deletes it, installs the runner, and registers.

A separate scale-down Lambda runs on a schedule — every five minutes by default — walks the instances, and terminates the ones that are not busy, deregistering them from GitHub on the way out. It is a poll, not a signal. The module's own documentation is candid that this is brute force.

There is one more Lambda that looks like an odd inclusion until you hit the problem it solves. Downloading the runner distribution from GitHub is occasionally very slow — minutes, not seconds. So a syncer copies the release into an S3 bucket on a schedule, and instances fetch it from there.

The getting-started guide walks the setup in three phases: create a GitHub App, apply the Terraform, then come back and point a webhook at the API Gateway URL that Terraform printed. The two-pass shape is unavoidable — each side needs an output from the other.

One deployment, and why that was not enough

The first version was a single deployment for the whole organisation. It worked, and it was wrong within a fortnight.

The failure is easy to describe. Every queued job in every repository goes onto one SQS queue and competes for one runners_maximum_count. One product's integration suite fans out into a wide matrix, saturates the ceiling, and a one-job deploy for an unrelated service sits behind it. Raising the ceiling raises it for everyone, including the workload that was already the problem. The isolation you want is not a bigger number; it is a different queue.

So we stopped deploying the module once. We deployed it once per product, each with its own prefix, its own webhook, its own queue, its own ceiling, and its own label.

module "runners_billing" {
  source  = "philips-labs/github-runner/aws"
  version = "1.9.0"

  aws_region = local.aws_region
  vpc_id     = module.vpc.vpc_id
  subnet_ids = module.vpc.private_subnets

  prefix = "ci-billing"
  tags   = { Product = "billing" }

  github_app = {
    id             = var.github_app_id
    key_base64     = var.github_app_key_base64
    webhook_secret = random_id.billing_webhook_secret.hex
  }

  webhook_lambda_zip                = "lambdas-download/webhook.zip"
  runners_lambda_zip                = "lambdas-download/runners.zip"
  runner_binaries_syncer_lambda_zip = "lambdas-download/runner-binaries-syncer.zip"

  enable_organization_runners = true
  runner_extra_labels         = "billing"

  runner_enable_workflow_job_labels_check     = true
  runner_enable_workflow_job_labels_check_all = true

  instance_types                = ["m5.large", "m5a.large", "c5.large"]
  instance_target_capacity_type = "spot"
  instance_allocation_strategy  = "capacity-optimized"

  runners_maximum_count           = 8
  minimum_running_time_in_minutes = 5
  delay_webhook_event             = 10
}

A second module block, runners_search, is the same thing with prefix = "ci-search", runner_extra_labels = "search", and whatever ceiling that product's builds actually need. The blocks are boring and nearly identical, which is the point: the interesting part is that they do not share a queue.

The workflow asks for its own product's runners by label:

jobs:
  test:
    runs-on: [self-hosted, linux, x64, billing]

Sizing is per product too. The service with a heavy test suite got larger instance types and a ceiling of sixteen. A service whose pipeline is a lint, a unit run and a container push got two t3.mediums and never noticed. Under one deployment, both of those decisions have to be the same decision.

The label check is not optional here

This is the part I would have got wrong without reading the variable descriptions, and it is the difference between the arrangement above working and it being an expensive mess.

We used one organisation-level GitHub App for the API credentials, and a separate organisation webhook per deployment, each pointed at its own API Gateway URL with its own secret. GitHub does not route between them. Every workflow_job event goes to every webhook. Each deployment's webhook Lambda sees the billing job, the search job and everything else.

By default the module accepts those events. Every deployment would start an instance for every queued job, and the label the job asked for would only be consulted later, when the runners tried to claim work. You would pay for six fleets to answer one job and then watch five of them idle until the scale-down poll came round.

runner_enable_workflow_job_labels_check moves that decision to the webhook. With it on, the Lambda compares the labels on the event against the labels this deployment's runners will actually carry — self-hosted, the OS, the architecture, plus whatever is in runner_extra_labels — and drops the event if they do not line up.

runner_enable_workflow_job_labels_check_all decides how strict that comparison is. It defaults to true, and true is what you want: every label on the job must be one this deployment offers. Set it to false and a match on any single label is enough — which means self-hosted, a label every one of these jobs carries, matches every deployment, and you are back to the fan-out. I set both explicitly rather than leaning on the default, because the default being right is not the same as the next person knowing it is load-bearing.

repository_white_list is the belt to that pair of braces. Naming the repositories a deployment is allowed to serve means a mislabelled workflow in some other repository cannot quietly start machines on the billing product's budget.

What zero actually costs

The runners scale to zero. The infrastructure around them does not.

Our floor was the plumbing: NAT gateways so instances in private subnets can reach GitHub and S3, API Gateway, the Lambdas, the S3 bucket holding the runner distribution, Parameter Store, and log retention. A NAT gateway is charged by the hour whether or not anything is behind it, which works out to roughly $30 a month each before data processing, and it was comfortably the largest line that did not move. Everything else in that list was cents. Roughly half of our monthly total was this floor, and it would have been the same on a month with no builds at all.

The other half tracked build minutes, and this is where spot earns its keep. instance_target_capacity_type = "spot" with capacity-optimized allocation across three instance types meant we very rarely paid on-demand and very rarely got interrupted. Call it $90 a month of compute at spot rates for an m5.large-shaped fleet: somewhere around two thousand runner-hours.

Two thousand hours of the same work on GitHub-hosted Linux runners, at $0.008 per minute past the included allowance, is a little over a thousand dollars. That gap is the entire argument, and it is not a close one at this volume.

The comparison flatters us slightly and I would rather say so. Hosted minutes are billed per job, rounded up. Our instance-hours include the boot, and they include however long a runner sat idle before the five-minute scale-down poll found it. minimum_running_time_in_minutes deliberately makes that worse in exchange for reusing a warm machine. The real ratio is not ten to one. It was still not close.

The line that surprised me was CloudWatch Logs. Streaming runner logs off the instances is genuinely useful when a build fails on a machine that no longer exists, and it is also the one component whose cost scales with how chatty your builds are rather than with how long they run. Setting a retention period was a one-line fix I should not have needed prompting to make.

What I still own

Terraform owning the fleet is not the same as nobody owning it.

Jobs start slower. A cold runner is an EC2 launch, a user-data script, and a registration, and that is a couple of minutes before the first line of the build. delay_webhook_event intentionally adds to it so an already-running runner gets first refusal. Where the wait mattered we used idle_config to keep a small warm floor during working hours, which is a direct trade of money for latency and should be made deliberately rather than by leaving the default.

The image drifts. The default Amazon Linux 2 AMI plus a user-data script is a fine starting point and a slow one, because every instance installs the same things again. Baking a prebuilt AMI moves that work to build time. It also means you now maintain an AMI pipeline, which is a real piece of work that did not exist when GitHub owned the machines.

Reusable or ephemeral is a real decision. Reusable runners take one job after another until they go idle, which is fast and means one job can leave state behind for the next. Ephemeral runners are destroyed after a single job, which is clean and pays the boot cost every time. We kept reusable runners where builds were frequent and cheap, and used ephemeral ones where a job handled deployment credentials. That split was worth the extra deployment.

Public repositories are off the table. A self-hosted runner will happily execute whatever a pull request tells it to, and on a public repository that is a stranger's code running inside your VPC. Everything on this fleet was private, and I would not relax that for convenience.

Spot instances go away. Rarely, with the diversified instance types, but it happens, and the job dies with the machine. Anything long enough for that to sting needs to be retryable on its own terms.

Where I would still not do this

None of the above is worth doing for a repository with a five-minute build. The included allowance covers it, the hosted runner is already patched, and the fixed floor here costs more per month than the minutes ever would. The crossover is real but it is not near zero.

What made it worth it for us was the shape of the workload rather than only its size: a set of products, deploying independently, on a network the hosted runners cannot see. Once you are on your own machines anyway, giving each product its own queue and its own ceiling costs one more module block, and it removes an entire category of complaint about someone else's tests holding up your release.

---

The module has since moved to the github-aws-runners organisation, and its Terraform registry path with it. The links above point at its current home; the source in the example is the one that was correct at the time of writing.