A month after we moved the database onto Atlas, the bug reports started arriving with a shape we could not answer. Not "the page is broken" — those are easy. The hard ones were "this one account's invoice totals are wrong", which is a bug that lives entirely in the data. Nobody can reproduce it from a seed script, because the seed script contains the data you thought of and the bug is in the data you did not.

There were three ways to go and two of them were bad.

Giving engineers read access to production is the fastest and it is the one I will not defend. It puts every customer's records one autocomplete away from an aggregation someone runs while tired, and no amount of "we only look at what we need" survives contact with an incident at 2am.

Building better fixtures is the answer everyone reaches for and it does not work for this class of bug. The fixture is a hypothesis about what production looks like. If the hypothesis were correct, the bug would not exist.

The third option is to take a real backup, restore it somewhere disposable, remove the customers from it, and hand an engineer the result. Atlas takes snapshots anyway, and its API can restore one into a cluster you create on demand. So the mechanics were all available. What took the thinking was the order.

The window nobody talks about

Here is the thing that shaped everything else.

Atlas's automated restore does not create a cluster for you. The target has to exist first — you create a cluster, then you create a restore job pointing at it, and Atlas overwrites the target's contents with the snapshot. Which means there is a period, between the restore job completing and the scrubber finishing, when a cluster exists, is running, and is full of real customer data.

If anything can connect during that window, the whole exercise is theatre. You have not built a safe copy of production; you have built a second production with worse access controls and a shorter memory.

So the design rule is a single sentence, and every other decision follows from it: nothing that a human controls exists until the scrub has passed. During that window the cluster has exactly one database user — the workflow's own — no IP access list entries beyond the runner, and no private endpoint. It is not that engineers are told not to connect. It is that there is nothing to connect with.

That inverts the natural order you would write if you were not thinking about it. The obvious workflow is: create cluster, add the requester's user, restore, scrub, post the connection string. It works, it is one step shorter, and it leaves a door open for twenty minutes.

The shape of the workflow

It is a workflow_dispatch, which is the right trigger because someone always has a reason and the reason should be typed into a box that gets logged.

on:
  workflow_dispatch:
    inputs:
      snapshot:
        description: 'Snapshot to restore (latest, or a snapshot id)'
        default: 'latest'
      ttl_hours:
        description: 'Destroy the cluster after this many hours'
        default: '8'
      reason:
        description: 'Ticket or incident this is for'
        required: true

jobs:
  provision:
    runs-on: ubuntu-latest
    environment: troubleshooting-clusters
    timeout-minutes: 90

The environment line is the approval gate. GitHub environments take required reviewers, so the run pauses until someone other than the requester approves it, and the approval is recorded next to the reason. That is most of the audit trail you want, for one line of YAML, and it is the sort of thing that makes GitHub Actions worth the switch rather than a lateral move.

The steps, in the order that matters:

1. Find the snapshot.
2. Create the target cluster. Wait for it to reach IDLE.
3. Create the restore job. Wait for it to finish.
4. Run the scrubber. Fail the whole workflow if it does not pass.
5. Only now: create the requester's database user and access-list entry.
6. Post the connection string to the requester, and record the TTL.

Step 4 failing means step 5 never runs, and the cleanup step tears the cluster down. A scrubber that cannot finish leaves you with no cluster, which is the correct outcome and took one if: failure() to arrange.

Talking to the Atlas API

The Atlas Administration API is HTTP with digest authentication, which is the first small surprise — it is not a bearer token, so curl needs --digest and anything you write in Node needs a client that can do the challenge-response round trip rather than one that sets an Authorization header once.

Finding last night's snapshot:

curl -s --digest --user "$ATLAS_PUBLIC_KEY:$ATLAS_PRIVATE_KEY" \
  "https://cloud.mongodb.com/api/atlas/v1.0/groups/$GROUP_ID/clusters/$SOURCE/backup/snapshots" \
  | jq -r '.results | sort_by(.createdAt) | last | .id'

Creating the cluster is a POST to /groups/{groupId}/clusters, and then you poll the same path until stateName is IDLE. There is no webhook and no long-poll; provisioning a dedicated tier takes several minutes and you wait for it. Then the restore:

curl -s --digest --user "$ATLAS_PUBLIC_KEY:$ATLAS_PRIVATE_KEY" \
  -X POST -H 'Content-Type: application/json' \
  "https://cloud.mongodb.com/api/atlas/v1.0/groups/$GROUP_ID/clusters/$SOURCE/backup/restoreJobs" \
  -d "{
        \"deliveryType\": \"automated\",
        \"snapshotId\": \"$SNAPSHOT_ID\",
        \"targetClusterName\": \"$TARGET\",
        \"targetGroupId\": \"$GROUP_ID\"
      }"

Note whose path that is. The restore job is created against the source cluster, and the target is a parameter of the body. I got that backwards on the first attempt and spent a while being told the cluster had no snapshots, which was true, because I was asking the brand-new empty one.

The two credentials this needs are an Atlas API key pair, and they are the one part of this that is not as tidy as I would like. Everything else in our CI authenticates to AWS with OIDC and holds no long-lived secrets; the Atlas key is a stored secret with real power over the project, scoped as narrowly as the project roles allow and rotated on a schedule. It is a genuine soft spot and I would not pretend otherwise.

The scrubber

This is the part that is actually mine, and it is a Node script rather than a shell pipeline for one reason: the rules are per-field, and per-field rules want a language.

The naive version replaces every email with a constant. It fails immediately, because there is a unique index on that field and the second document collides. That failure is a gift — it forces you into the design you wanted anyway, which is that every masked value must be derived from the document, deterministic, and unique where the original was unique.

Deriving from _id gets all three. It also preserves something you would otherwise lose without noticing: if the same customer's email appears in two collections, both copies mask to different values, and joins across them break. So the derivation is keyed on the identity the field refers to, not on the document it happens to sit in.

The work runs server-side. updateMany with an aggregation pipeline means the documents never travel to the runner, which matters when the collection is large and matters more when you remember what is in it:

await db.collection('members').updateMany(
  {},
  [
    {
      $set: {
        email: {
          $concat: [
            'member-',
            { $toString: '$_id' },
            '@scrubbed.invalid',
          ],
        },
        firstName: { $concat: ['Member ', { $substrCP: [{ $toString: '$_id' }, 18, 6] }] },
        lastName: 'Scrubbed',
        phone: null,
        // Kept: it drives the bug we are chasing and identifies nobody.
        // billingPlan, createdAt, status
      },
    },
  ],
)

.invalid is a reserved TLD that can never resolve, which means that if some half-configured job in the copy ever tries to send mail, it fails at DNS instead of reaching a person. That is a deliberately cheap safety net and it has caught things.

Some collections are not masked at all, they are dropped. Payment provider tokens, webhook request bodies, stored export files, the audit log's payload field — all of these are either useless for debugging or impossible to mask credibly, and a collection you drop cannot leak through a rule you got wrong.

The list that fails closed

The rule I am most glad about is the smallest.

A scrubber configured with a list of collections to clean is a denylist, and a denylist is wrong the moment someone ships a new collection. The person adding customer_notes in a feature branch is not thinking about a troubleshooting workflow they have never run.

So the script does not just process its list. It reads the collections that actually exist in the restored database, compares that against the set it knows about, and exits non-zero if there is anything it has not been told what to do with:

const present = new Set((await db.listCollections().toArray()).map((c) => c.name))
const known = new Set([...Object.keys(RULES), ...DROP, ...PASS_THROUGH])
const unknown = [...present].filter((name) => !known.has(name))

if (unknown.length > 0) {
  throw new Error(
    `Unclassified collections: ${unknown.join(', ')}. ` +
      `Add them to RULES, DROP or PASS_THROUGH in scripts/scrub.js.`,
  )
}

PASS_THROUGH is explicit and that is the point — a collection is safe because somebody wrote down that it is safe, not because nobody mentioned it. The first time this fired, it was on a collection that had been added six weeks earlier and held free-text notes staff write about accounts. Under a denylist it would have been copied verbatim into a cluster three engineers had the password to.

The cost is that the workflow occasionally fails for a reason that is not the requester's fault, and someone has to go and classify a collection. That has happened perhaps five times. Each of those five is a small annoyance in exchange for the one that mattered.

What it cost

A dedicated cluster large enough to hold a production snapshot, for the eight hours the default TTL allows, is a few dollars — call it the price of a round of coffee per investigation. The TTL is enforced by a scheduled workflow that runs hourly, lists clusters carrying a deleteAfter tag, and deletes the expired ones, because a TTL that depends on the original job still running is not a TTL.

The wall-clock cost is more interesting than the money. Cluster provisioning plus restore plus scrub was around thirty-five to fifty minutes end to end for us, almost all of it waiting on Atlas. That is fine for an investigation and far too slow to be part of anyone's inner loop, which shaped how it got used: people kicked it off, went and did something else, and came back to a connection string.

Where it does not help

Scrubbed data does not reproduce every bug. If the defect depends on the actual bytes in a customer's address field — an encoding problem, a length boundary, a name with a character your parser dislikes — masking has destroyed the evidence by construction. We hit this. There is no clever fix; the honest answer is that this tool is for bugs in relationships and volumes, and a different, much more carefully supervised process is for the other kind.

This is pseudonymisation, not anonymisation. The masked values are derived, deterministic, and preserve linkage across collections — that is exactly what makes the copy useful, and it is also exactly what stops it being anonymous data. The record structure survives, the graph survives, and enough of the shape survives that a determined person with outside knowledge could re-identify rows. I would not describe these clusters as outside the scope of data protection obligations, and neither should you. They are a real reduction in exposure and a genuinely smaller blast radius. They are not a magic word that makes the data stop being about people.

The scrubber is a security control that looks like a script. A typo in a field name silently leaves a column unmasked and everything still exits zero. The unknown-collection check catches new collections, not new fields, and I never closed that gap properly. If I were doing it again, the assertions would be part of the script's own output — count the documents where a masked field still matches an email pattern, and fail on any of them — rather than trusting the rules to have been written correctly.

That last one is the honest summary of the whole thing. The Atlas API calls took an afternoon. Deciding what could be allowed to exist, and in what order, took considerably longer and is the only part I would carry to a different company.

---

Written against the Atlas Administration API as it stood in late 2023. The paths have since moved to /api/atlas/v2 with a versioned Accept header, and API keys are now the legacy authentication method alongside OAuth service accounts. The restore semantics — target must exist, target gets overwritten — are unchanged, and they are the part this design hangs on.