Fifteen months ago I wrote about choosing BullMQ over Agenda for the accounts service, and I still think that was the right call for what the service was at the time: one producer, one consumer, one region, and a customer staring at a spinner. None of those four things is true any more, and the thing that broke first was not the one I would have guessed.

It was not latency. Blocking pops are still faster than anything I am about to describe. What broke was that queue.add('sync-seats', payload) names its consumer. The producer has to know that seat syncing is a job, that the job lives on the billing queue, and that some worker somewhere is reading it. When a second team wanted to know that a subscription had changed, the only place to put that knowledge was in the producer, and the producer was already carrying four other people's requirements.

The second thing that broke was geography. We run in Ireland, Singapore and Sydney. A Redis instance is a regional thing, so a queue is a regional thing, so anything that has to be told about a change in Singapore from a process running in Ireland is not a queue problem any more. It is a networking problem wearing a queue's clothes.

This post is the infrastructure half of getting out of that. The client library that sits on top of it is its own post.

The shape

One topic per service. The service owns it, publishes facts about itself onto it, and does not know or care who is listening. One queue per worker, and a worker subscribes its queue to whichever topics it needs. Adding a consumer is a subscription; it is not a change to anybody's producer.

We already had a shape for the compute, from the service catalog work: two container types behind two Proton templates. A web service sits behind an ALB, handles requests and publishes. A worker has no load balancer, no ingress and no port; it gets a queue, and it scales on how far behind that queue is. Workers publish too — the long-running things spawn more work — so the publish path is not a web-server-only concern.

That gives a fan-out with three regions in it and, on the day we drew it, eleven queues. It also gives you eleven places for a message to stop, which is the honest cost and I will come back to it.

Every topic is FIFO, and the reason is not ordering

The default in our topic module is fifo_topic = true, and that decision has consequences that ripple all the way into the SDK.

The mechanical ones first. A FIFO topic can only deliver to a FIFO queue, so the queue module defaults to FIFO too. Both names have to end in .fifo, which means the name is not just cosmetic and every module that composes an ID has to know about the suffix. Every published message needs a MessageGroupId, and either a MessageDeduplicationId or content-based deduplication turned on at the topic.

Now the honest part. Our SDK's default is a fresh UUID for both the group ID and the deduplication ID on every message. A unique group per message means there is exactly one message in each group, which means there is nothing to order. A unique deduplication ID means nothing will ever be recognised as a duplicate. Out of the box, we get neither of FIFO's two features.

So why pay for it. Because the topic type is fixed at creation. You cannot turn a standard topic into a FIFO topic later; you delete it, create a new one, and recreate every subscription pointing at it, across three regions, while producers are publishing. A service that will one day need accountId as its message group — and billing will, the first time two seat changes land out of order — can start using one by passing an option. A service that started standard has to schedule a migration.

FIFO by default is not a claim that we need ordering. It is a door we paid a small amount to keep open, and I would rather say that plainly than pretend the queue is ordered when the group IDs say it is not.

The one place we do take the throughput settings seriously is the queue:

resource "aws_sqs_queue" "main" {
  name       = "${local.id}${var.fifo ? ".fifo" : ""}"
  fifo_queue = var.fifo

  # Both of these have to be at message-group level or the queue is capped at
  # the low per-queue FIFO quota, and a fan-out target is exactly the thing
  # that will hit it.
  deduplication_scope   = var.fifo ? "messageGroup" : null
  fifo_throughput_limit = var.fifo ? "perMessageGroupId" : null

  # Job processors are the slow half of the estate; a two hour lease is not
  # generous for them, it is realistic.
  visibility_timeout_seconds = var.visibility_timeout_seconds
  message_retention_seconds  = var.retention_days * 24 * 60 * 60
  sqs_managed_sse_enabled    = true
}

The subscription belongs to the topic's region

This is the part I expected to be hard and it was, just not for the reason I thought. SNS will deliver across regions. The subscription resource, though, is a thing that exists in the topic's region, not the queue's, so Terraform has to create it through a provider aliased to wherever the topic lives.

The worker template takes a flat list of topic ARNs from the service form, buckets them by the region embedded in the ARN, and creates one set of subscriptions per region:

locals {
  by_region = {
    for region in ["eu-west-1", "ap-southeast-1", "ap-southeast-2"] :
    region => {
      for arn in var.subscribe_to : arn => arn
      if split(":", arn)[3] == region
    }
  }
}

resource "aws_sns_topic_subscription" "dub" {
  for_each = local.by_region["eu-west-1"]
  provider = aws.dub

  topic_arn = each.value
  protocol  = "sqs"
  endpoint  = module.queue.arn

  # Without this, the body the consumer receives is an SNS envelope with the
  # real payload as a JSON string inside it, and the message attributes are
  # inside the envelope rather than on the SQS message. The consumer reads one
  # of those attributes before it decides how to parse the body, so raw
  # delivery is not a preference here. It is load-bearing.
  raw_message_delivery = true
}

Then the same block twice more for aws.sin and aws.syd. It is repetitive and it cannot be a for_each over providers, because provider references are not values. Three near-identical blocks is the price, and every time somebody adds a region they will forget the fourth one. I do not have a better answer than a comment and a test.

The subscribe_to input is validated against the ARN shape, which sounds pedantic until you have watched a plan succeed against a typo and produce a queue that nothing publishes to:

variable "subscribe_to" {
  type    = list(string)
  default = []

  validation {
    condition = alltrue([
      for arn in var.subscribe_to :
      can(regex("^arn:aws:sns:(eu-west-1|ap-southeast-1|ap-southeast-2):[0-9]{12}:[a-zA-Z0-9_-]+(\\.fifo)?$", arn))
    ])
    error_message = "Each entry must be an SNS topic ARN in a supported region."
  }
}

Deny by default, on both ends

Here is where the effort actually went. Not the fan-out — the fan-out is three resources. The policies.

A topic with no resource policy is not open to the world, but it is open to the account, and the account contains every task role we run. The intent we wanted was narrower: only the roles belonging to services in this environment and this stage may publish here. And an allow-list is the wrong instrument for that, because allow-lists compose with every other allow you have, including the account-level ones you did not write. An explicit deny is the only statement that cannot be out-voted.

data "aws_iam_policy_document" "topic" {
  statement {
    sid       = "DenyPublishFromOutsideThisStage"
    effect    = "Deny"
    actions   = ["sns:Publish"]
    resources = [aws_sns_topic.main.arn]

    principals {
      type        = "AWS"
      identifiers = ["*"]
    }

    condition {
      test     = "StringNotLike"
      variable = "aws:PrincipalArn"
      values = compact([
        # Task roles are named "<env>-<stage>-<service>-task" by the service
        # template, so the convention is what makes this expressible at all.
        "arn:aws:iam::${local.account_id}:role/*-${local.stage}-*-task",

        # The scheduler needs to publish delayed messages on someone's behalf.
        "arn:aws:iam::${local.account_id}:role/${local.env}-${local.stage}-scheduler-publish-role",
      ])
    }
  }
}

Read that carefully, because I got it wrong twice. StringNotLike inside a Deny means deny everyone whose principal ARN does not match one of these patterns. Flip the condition operator and you have denied exactly the callers you meant to allow, and the failure arrives as an AuthorizationError from a publish call in a service you were not deploying.

The queue end needs two separate ideas expressed:

• Nothing may SendMessage to this queue except the topics it is subscribed to. Not other topics, not services with a queue URL and good intentions. That one keys on aws:SourceArn, which for an SNS delivery is the topic.
• Nothing may ReceiveMessage except the task role of the worker that owns it. A queue with two readers is a queue where half your messages disappear into a process that was not written to handle them.

Both are deny statements for the same reason as above.

What makes all of this work is the naming convention, and that is worth being uncomfortable about. *-${stage}-*-task is a policy written against a string pattern that some other module is responsible for producing. Nothing enforces the link. If somebody creates a task role by hand and names it something sensible but different, they are silently outside the fence — or, worse, if somebody names an unrelated role to match the pattern, they are silently inside it. Tags with a condition on aws:PrincipalTag would be the better instrument. We did not have consistent tagging on roles yet, and I did not want the messaging migration to also be the tagging migration.

The dead-letter queue, and a redrive count of one

Every queue gets a DLQ, a redrive policy pointing at it, and a redrive allow policy on the DLQ naming the source queue so nothing else can adopt it.

The default maxReceiveCount is 1. That surprises people, because the received wisdom is to retry a few times before giving up. The reasoning:

The visibility timeout is two hours. A message that fails and is left to time out is invisible for two hours before anyone can look at it again. With a receive count of three, a poison message is a six-hour round trip before it lands somewhere a human will find it, and for six hours the only symptom is a queue that is quietly getting older.

Retries are also the wrong layer. A transient AWS error inside a handler is something the handler's own client should retry, in seconds, with the context still in memory. A failure that survives that is not transient, and running the whole handler again two hours later against a job that may not be idempotent is not a recovery strategy, it is a second incident.

So: one attempt, then the DLQ, and the DLQ has a fourteen-day retention and an alarm on it. The SDK does something on top of this that makes the two hours mostly theoretical, which is the other post's problem.

Scaling on age, not on depth

The obvious autoscaling signal for a worker is queue depth, and it is the wrong one. Depth tells you how much work is waiting; it does not tell you whether anybody is getting to it. A thousand messages that drain in ten seconds and ten messages stuck behind a slow handler look nothing alike on ApproximateNumberOfMessagesVisible and identical to a customer.

ApproximateAgeOfOldestMessage answers the question we actually care about: how late is the latest thing. Two alarms, one on each side of the band, wired to the step scaling policies:

resource "aws_cloudwatch_metric_alarm" "behind" {
  alarm_name          = "${local.id}-queue-age-high"
  namespace           = "AWS/SQS"
  metric_name         = "ApproximateAgeOfOldestMessage"
  statistic           = "Maximum"
  comparison_operator = "GreaterThanThreshold"
  threshold           = var.scale_up_age_seconds
  period              = 60
  evaluation_periods  = 1
  dimensions          = { QueueName = module.queue.name }
  alarm_actions       = [aws_appautoscaling_policy.up.arn]
}

The scale-down cooldown is five times the scale-up cooldown, because the cost of being one task too big for five minutes is a rounding error and the cost of flapping is a worker that spends its life in PROVISIONING.

The thing BullMQ had that SNS does not

Delay. queue.add(name, data, { delay: 900_000 }) is one option object, and there is no equivalent anywhere in SNS or SQS. SQS has a delay, but it is capped at fifteen minutes, and the things we needed to postpone were measured in hours and days — a trial ending, a grace period expiring, a reminder before a renewal.

The answer is EventBridge Scheduler: a one-time schedule per delayed message, in a schedule group per environment, created with ActionAfterCompletion set to DELETE so it removes itself once it fires and does not accumulate against the account quota.

Two details that cost an afternoon each. The first is that you want the universal target — arn:aws:scheduler:::aws-sdk:sns:publish — rather than the topic ARN directly, because the direct target sends the schedule's input as the message body and nothing else. The universal target lets you hand it a complete publish request, so the message attributes and the FIFO group and deduplication IDs survive the trip and the consumer cannot tell a delayed message from a direct one. The second is that the scheduler publishes under its own role, which is why that role is in the topic's deny exception list above — and why a service that creates a schedule and then gets an AuthorizationError fifteen minutes later is debugging an IAM problem with no stack trace attached.

What it cost

A message used to be: LPUSH, BRPOPLPUSH, handler. Two hops, one datastore, and redis-cli for when it went wrong.

It is now: publish to a topic, a subscription filters and forwards, delivery to a queue in possibly another region, a poll, a handler, and a delete. Five places for it to stop. There is no Bull Board equivalent — there is CloudWatch, where a queue looks like four metrics and a message looks like nothing at all. The operational signal is not "look at the queue", it is "the DLQ is non-empty and the oldest-message age is climbing", and getting the team to read those two numbers instead of asking for a UI took longer than writing any of the Terraform.

What we got back: adding a consumer no longer touches a producer, the regions talk to each other without anybody terminating TLS on purpose, and Redis is a cache again — which, as I wrote in the BullMQ post, is what most Redis installations were configured to be all along.

---

Regions are eu-west-1, ap-southeast-1 and ap-southeast-2. Everything above is provisioned through Proton service templates, which open a pull request against your repository rather than applying anything, so all of it lands as a reviewable diff.