We were running MongoDB ourselves — a replica set on EC2 instances, built by people who had moved on and patched by people who had not volunteered. It worked. The argument for moving it onto Atlas was never that it was broken; it was that the on-call rotation included a database nobody currently on the team had chosen, and every upgrade was a small project.

The constraint was that we could not stop. Not "we would prefer not to" — the platform is used across time zones, and there is no hour of the week where a maintenance window is free. So the migration had to happen underneath a running application, and the application had to keep writing the whole time.

Two things made that possible, and only one of them is interesting. The uninteresting one is that mongomirror does exactly this job and does it well. The interesting one is that all of the risk had moved into the network long before any data did.

The network came first

The default way to let something reach an Atlas cluster is the IP access list: you tell Atlas which public addresses may connect, and traffic goes over the internet with TLS on top. It is not insecure and I am not going to pretend it is. It was still the wrong shape for us, because the list is a list — it needs an entry per NAT address, it drifts when the network changes, and the answer to "can this thing reach the database" becomes a question about a text field in a web console rather than about the VPC.

VPC peering is the obvious next step and we skipped it too. Peering joins two networks; we did not want to join anything, we wanted one directional door. Peering also means caring about CIDR overlap forever, and it exposes rather more of Atlas's network to ours than the job required.

AWS PrivateLink gives you the door. Atlas publishes an endpoint service, you create an interface endpoint in your own VPC, and the cluster becomes reachable at a private address inside your subnets. Nothing traverses the internet, there is no route between the two VPCs, and the security question collapses into a security group — which is a thing Terraform already owns.

The Terraform shape is a three-step chain, and the order is forced by which side knows what:

resource "mongodbatlas_privatelink_endpoint" "atlas" {
  project_id    = var.atlas_project_id
  provider_name = "AWS"
  region        = "EU_CENTRAL_1"
}

resource "aws_vpc_endpoint" "atlas" {
  vpc_id             = module.vpc.vpc_id
  service_name       = mongodbatlas_privatelink_endpoint.atlas.endpoint_service_name
  vpc_endpoint_type  = "Interface"
  subnet_ids         = module.vpc.private_subnets
  security_group_ids = [aws_security_group.atlas_endpoint.id]
}

resource "mongodbatlas_privatelink_endpoint_service" "atlas" {
  project_id          = mongodbatlas_privatelink_endpoint.atlas.project_id
  private_link_id     = mongodbatlas_privatelink_endpoint.atlas.private_link_id
  endpoint_service_id = aws_vpc_endpoint.atlas.id
  provider_name       = "AWS"
}

Atlas has to create the endpoint service before AWS can be told what to connect to, and AWS has to create the interface endpoint before Atlas can be told to accept it. Terraform resolves that on its own through the references, which is the entire reason to do it in Terraform rather than by hand in two browser tabs. Where a human would tab back and forth pasting identifiers, the graph just runs.

Two details cost me time, and neither is guessable.

The port range is not 27017. Behind a private endpoint, Atlas puts a load balancer in front of the replica set and gives every node in the region its own port on it — 1024, 1025, 1026 and upward — because all the nodes now answer on one hostname. So the security group cannot be a tidy 27017 rule. Clients need egress to the endpoint across 1024–65535, and the endpoint's own group needs matching ingress. The first time you write the sensible-looking rule, DNS resolves, the TCP connection to the first node succeeds, and the driver then fails to complete topology discovery because it cannot reach the other two. It looks like an Atlas problem and it is a security group problem.

The connection string is a different string. A cluster behind a private endpoint has its own mongodb+srv:// URI, and it is not the one on the cluster's overview page. In Terraform it comes off the cluster resource under connection_strings.private_endpoint, and it does not exist until the endpoint service is actually established — so the cluster resource needs an explicit depends_on pointing at mongodbatlas_privatelink_endpoint_service, or the first apply will hand you an empty list and the second will quietly fix it. Implicit dependency is not enough here because nothing in the cluster's arguments refers to the endpoint.

By the time any data moved, the network had been applied, destroyed and re-applied several times against a throwaway project. That was the point of doing it first: on migration day, "can we reach Atlas" was not an open question.

Where mongomirror has to run

mongomirror connects to both databases at once. It does an initial sync of every collection, then tails the source oplog and applies the changes to the destination until you tell it to stop. That means it needs a vantage point that can see the source replica set and the Atlas cluster.

After PrivateLink, exactly one kind of place satisfies that: an instance inside the VPC, in a subnet the interface endpoint lives in, with a security group allowed through it. Not a laptop, not a bastion outside the private subnets, not a CI runner on hosted infrastructure. The network decision made in the previous section quietly decided this one too.

So it ran on a plain EC2 instance in the same VPC and the same private subnets as the endpoint. Three things about that box mattered:

Network, not CPU. The initial sync is bounded by how fast you can pull documents out of one side and push them into the other. We gave it a network-optimised instance and never saw the CPU become the constraint.

Disk, because of the oplog. mongomirror needs the source's oplog window to still cover the whole of the initial sync when the sync finishes. If the sync takes longer than the oplog holds, the tail has nothing to resume from and you start again. --oplogPath buffers oplog entries to local disk while the sync runs, which converts "is our oplog big enough" into "is this volume big enough", and the second question has a much better answer. The sizing rule is unglamorous: if the source oplog holds a day and the sync takes two, you want roughly two days' worth of oplog in free space.

It has to survive your SSH session. This runs for hours. It ran under systemd with its output to the journal, because the first thing that goes wrong on a long-running foreground process is a laptop lid.

Permissions are two users, and they are not symmetric. On the source, the user needs the built-in backup role — read any database, plus read local, which is where the oplog is. On the Atlas side it needs Atlas admin, because it is recreating collections and indexes rather than just writing documents.

The command

mongomirror \
  --host "rs0/mongo-a.internal:27017,mongo-b.internal:27017,mongo-c.internal:27017" \
  --username migrator \
  --password "$SOURCE_PASSWORD" \
  --authenticationDatabase admin \
  --ssl \
  --sslCAFile /etc/ssl/internal-ca.pem \
  --destination "atlas-prod-shard-0/pl-0-eu-central-1.abcde.mongodb.net:1024,..." \
  --destinationUsername migrator \
  --destinationPassword "$ATLAS_PASSWORD" \
  --numParallelCollections 8 \
  --oplogPath /mnt/oplog \
  --bookmarkFile /mnt/state/mongomirror.bookmark \
  --httpStatusPort 8000

--host wants the replica set name and its members, not a URI. --destination wants the same shape for Atlas, and this is where the private endpoint shows up concretely: those are per-node ports on one load-balanced hostname, which is the port-range business from earlier arriving in a command line.

The flags I would argue about:

--numParallelCollections defaults to 4. Raising it helps when you have many collections of similar size and does nothing when one collection is most of your data, which is the more common shape. Ours was the common shape. Eight was a guess that made the small collections finish sooner and left the big one exactly as slow as it was going to be.

--bookmarkFile is what makes the run resumable. It records how far the oplog tail has got. Put it somewhere that outlives the process — the point of it is to be there after a crash, and the default lands next to wherever you happened to be standing.

--drop we did not use. It drops user collections on the destination before copying, which is right for a target you are sure about and a loaded gun on a target you are not. Ours was a fresh cluster with nothing to drop, so the flag would only have been there to save us from a mistake we would rather not have made in the first place.

--noIndexRestore we also did not use, and I want to flag it because it is the one I would reach for on a much larger dataset. Index builds on the destination are a real part of the wall-clock time. Skipping them and building afterwards lets you control when that cost lands. It also means the destination is not actually ready when mongomirror says the sync is done, which is a footgun if anyone else is watching the same status endpoint and drawing conclusions.

Watching it on a port

--httpStatusPort is the flag that turned this from an anxious afternoon into a boring one. It exposes a small HTTP endpoint on the instance that reports what phase the process is in and how far behind the oplog tail is.

That is enough to build the only two things anyone actually wanted:

curl -s localhost:8000/status | jq '{phase, lag: .details.lagTimeSeconds}'

A one-line poll into Slack every few minutes during the initial sync, so nobody had to ask. And an alert on the lag going up rather than down once the tail started, because a lag that grows means the source is writing faster than the mirror can apply, and that is the one failure mode where waiting longer does not help.

The lag number is also the thing the cutover plan is built around, so it is worth having a real feel for it well before you need it. Ours settled at a couple of seconds and stayed there, which is what told us the cutover would be short.

The cutover

The sequence was written down, rehearsed against a copy, and executed in about four minutes.

1. Confirm the mirror is in oplog-tail phase and the lag has been in the low seconds for hours, not minutes.
2. Stop writes. For us that meant putting the API into a mode where write endpoints return a retryable error and the background workers stop consuming. Reads carried on being served from the old replica set the entire time.
3. Watch the lag go to zero. This is the step the whole design is for: with writes stopped, the tail has a finite amount of work left, and you can see it finish rather than assume it has.
4. Stop mongomirror.
5. Roll the application onto the new connection string and let the deployment replace the instances.
6. Turn writes back on.

Step 5 is the title of this post, and it is deliberately anticlimactic. The connection string lived in one place — a parameter the deployment reads, not a value baked into an image or repeated across services — so the migration's final act was changing one string and letting the normal deployment mechanism do the normal thing. If that value had been in eleven places, this would have been a different and much worse story, and the work to consolidate it happened weeks earlier for exactly this reason.

The old replica set stayed running, untouched and still receiving nothing, for a week. Rolling back would have meant reversing steps 5 and 6 and accepting the loss of whatever had been written to Atlas in the meantime — which is to say the rollback was real for about the first ten minutes and theatre after that. Having it be honestly real for ten minutes was worth the week of instance cost.

What "no downtime" actually meant

The application never stopped serving. There was no maintenance page, no scheduled window, and nobody was told to avoid the product that evening. That is what we meant when we said no downtime and I think it is the meaning that matters.

It is not the same as claiming nothing happened. There was a window of under a minute where writes were refused with a retryable status and the clients retried into it. A user who submitted a form at the wrong moment saw a spinner for a few seconds. A background job that would have run then ran slightly later. If your definition of zero downtime does not permit a brief write pause, then this was not zero downtime, and I would rather say that plainly than let a good headline number do work it has not earned.

The alternative — a genuinely pauseless cutover with dual writes or a proxy in front of both databases — is possible and is a substantially larger project. For a sub-minute write pause absorbed by retries that already existed, it was not close to worth it.

What I would tell myself in September

Do the network as its own piece of work, with its own apply and its own destroy. Everything I got wrong about PrivateLink, I got wrong in a scratch project against a cluster with no data in it. That is the cheapest possible place to discover the port range.

Consolidate the connection string before you need to change it. The migration was easy because there was one place to edit. That was not luck and it was not free.

Rehearse against a restored copy, timed. We ran the whole sequence against a cluster restored from a backup, including the write-pause, and it is the only reason anyone was willing to say "about four minutes" out loud. The rehearsal also caught that our write-pause mode did not actually stop one background consumer, which would have been an unpleasant thing to learn live.

Watch the lag, not the clock. Every version of this plan that goes wrong goes wrong by someone deciding enough time has probably passed. The status port exists so that nobody has to decide that.

The database is now somebody else's operational problem, which was the entire point, and the network in front of it is a hundred lines of Terraform that the next person can read.

---

mongomirror has since been retired. MongoDB's current paths for this are the Atlas Live Migration service and mongosync; the mechanics of the cutover — a sync, a tail, a lag you watch to zero — are the same, and so are the PrivateLink and connection-string parts, which is most of what this post is about.