Somebody wants a revenue dashboard. The revenue lives in a document database inside a VPC, reachable through a private endpoint, with no public listener and no route in from anywhere. The tool that is going to draw the dashboard is somebody else's SaaS, and so is the thing that will load the data into the warehouse it reads from.

There are two shapes available. Either the vendor comes in — a bastion, an inbound rule, an allow-list of their egress addresses that they change when they feel like it — or we go out.

We went out, and I still think that was the right call, but the reasons are more interesting than "outbound is safer" and the bill is bigger than it looks.

Why a connector at all

The obvious objection first: the database already emits change streams, and the platform already consumes them for product features. Why pay someone to read a database we can read ourselves?

Because a warehouse feed is not the same job as a product feature. A product feature cares about what changed in the last few seconds and is allowed to be lossy at the edges. A warehouse feed has to do a full historical backfill without knocking the cluster over, resume from a definite position after an outage, notice that a field started appearing in documents halfway through last year, and re-sync a collection from scratch when somebody decides the transformation upstream was wrong. Every one of those is a week, and the last one is a week that happens repeatedly.

For a platform team that is not a data platform team, buying that is the cheaper answer. The part we still own is the part below — getting the vendor to the database at all.

Reverse, not forward

A managed ingest connector will normally offer to reach a private database over SSH. The default reading of that is forward: they hold a key, they connect to a bastion we expose, they hop from there to the database.

That is a door. It has an inbound security-group rule, a public listener that exists whether or not a sync is running, and an authentication decision made by our host about a connection initiated by someone else's host. Every one of those is a thing to get wrong, and the allow-list is maintained by the party on the other side.

The other direction is a phone call. We run ssh -R outwards to a host the vendor operates, using an account they issue us, and publish the database's port on their side of the connection:

our VPC                            vendor
┌──────────────────────┐          ┌──────────────────┐
│ tunnel host ─────────┼── ssh ──►│ :27018           │
│      │               │   -R     │    ▲             │
│      ▼               │          │    │             │
│ mongod :27017        │          │  connector reads │
└──────────────────────┘          └──────────────────┘

Nothing listens for them. There is no inbound rule, no public database endpoint, and no allow-list to keep in sync. If the relationship ends, we stop the container and the access ends with it — no coordination, no ticket, no waiting for someone else to remove a key.

What we hold instead is a private key that logs into their host. That is a credential worth protecting, but its blast radius is an SSH account on a vendor machine, not a foothold in our network.

The tunnel is a container

The whole mechanism is autossh in a container, written out by Terraform through a remote-exec provisioner:

services:
  reverse-tunnel:
    image: jnovack/autossh
    environment:
      - SSH_REMOTE_USER=<vendor account>
      - SSH_REMOTE_HOST=<vendor host>
      - SSH_REMOTE_PORT=22
      - SSH_TUNNEL_PORT=27018     # the port they connect to
      - SSH_TARGET_HOST=<resolved mongod host>
      - SSH_TARGET_PORT=27017
    restart: always
    volumes:
      - ./auto-ssh.key:/id_rsa
    dns: [8.8.8.8, 1.1.1.1]

Two details in there earn their place.

autossh, not ssh under a restart policy. restart: always handles a process that exits. It does nothing at all for the failure that actually happens, which is a TCP connection that is dead without being closed — a NAT table entry expiring, a middlebox silently dropping an idle flow. The SSH process is still running and still happy; nothing arrives. autossh exists precisely to detect that, by running its own traffic through the tunnel and restarting when the echo stops coming back. A supervisor watching the process is watching the wrong thing.

The pinned public resolvers. They are there so the container can resolve the vendor's hostname, and they are a small wart: the container is deliberately not using the VPC resolver, which means the one name it can't resolve is anything private. That is fine only because the target host is resolved elsewhere — which is the next problem.

The target is an SRV record, and we take the first one

Behind a private endpoint, the database is not a hostname. It is an SRV record, and the host it points at is the vendor's business, not ours: they rotate it, they replace members during maintenance, and the port is not necessarily the obvious one.

So the module resolves the record at plan time and pulls the target and port out of it:

data "dns_srv_record_set" "service" {
  count   = var.target_srv_record == null ? 0 : 1
  service = "_mongodb._tcp.${var.target_srv_record}"
}

locals {
  target_address = substr(
    data.dns_srv_record_set.service[0].srv[0].target,
    0,
    length(data.dns_srv_record_set.service[0].srv[0].target) - 1,
  )
  target_port = data.dns_srv_record_set.service[0].srv[0].port
}

The substr is trimming the trailing dot off the fully-qualified name, which is the sort of line that tells you exactly how it was discovered.

Look at the index, though. srv[0]. A replica set publishes one SRV record per member, and this picks one of them and pins the tunnel to it for as long as the container runs. Two things follow.

The first is that the tunnel points at a single node. When that node is replaced — and managed clusters replace nodes on their own schedule — the tunnel keeps dialling a name that no longer serves, and the fix is to run Terraform again. That is a maintenance chore wearing the costume of infrastructure-as-code, and it does not announce itself; the sync simply stops.

The second is that plan-time resolution puts DNS into the diff. Any plan, for any reason, can come back non-empty because the cluster reshuffled overnight. Half the time that is the system telling you something useful. The rest of the time it is noise in a plan you were reading for something else entirely.

Neither is fatal, and I would still resolve the record rather than hardcode a host. But "Terraform resolves DNS for you" is a sentence that sounds like a convenience and behaves like a coupling.

Three clusters, three ports, and a directory collision

There is more than one cluster to feed, and the remote port namespace belongs to the SSH account, not to us — so the ports are an allocation, not a configuration:

module "reverse-tunnel" {
  port_on_remote    = 27018
  target_srv_record = "<analytics cluster>"
  # ...
}

module "reverse-tunnel-hybrid" {
  port_on_remote    = 27118
  target_srv_record = "<second cluster>"
  # ...
}

The first version of the module wrote its compose file to one fixed path. The second tunnel overwrote the first's, and docker-compose up -d --force-recreate in that directory then cheerfully replaced the running service with the new one. Two tunnels declared, one tunnel running, no error anywhere — the feed that stopped was simply the one whose Terraform ran first. Keying the project directory by the remote port fixed it, and the shape of the bug is worth remembering: a module that is not parameterised on disk is not reusable no matter how many variables it takes.

The other one that cost an afternoon: the key file written by the file provisioner had no trailing newline, and ssh rejected it outright. The fix is echo '' >> the key, which lives in the code with a comment, because nobody would believe it otherwise.

Where the tunnel lives

On the VPN instance. It already existed, it already had a public address and a foot in the VPC, it already ran Docker, and it is a t3a.nano doing almost nothing. Adding a container to it cost nothing and shipped that week.

The honest version is that one instance is now both the path every engineer uses to reach the environment and a load-bearing piece of the data pipeline. Reboot it for the first reason and you break the second. We wrote that down and did not fix it, which is the usual life cycle for a decision like this one.

The VPN underneath

Since the same box carries both, the VPN configuration is worth the detour — it has two choices in it that I would repeat and two I would not.

It is a split tunnel, deliberately. The client profile generator strips the line that would send all traffic over the VPN and appends an explicit route for the VPC:

echo "route $(cidrhost $vpc_cidr 0) $(cidrnetmask $vpc_cidr)" >> $user.ovpn
sed -i '/redirect-gateway def1/d' $user.ovpn

An engineer's entire internet should not transit a nano instance in one region, and the failure mode of a full tunnel is that the VPN becomes load-bearing for work that has nothing to do with the VPC.

It pushes the VPC resolver, read from the instance's own resolv.conf. Without it the client gets a route and no names — private endpoints resolve to nothing, and the symptom is a connection that hangs rather than an error that explains itself. This is the single most common way a split tunnel is shipped broken.

The access check is a name whitelist evaluated at TLS time. A tls-verify script greps the certificate's common name against a file:

/bin/grep -q "^`expr match "$3" ".*CN=\([^,]*\)"`$" "$1" && exit 0

Revocation is what a CRL is for, and a whitelist is a cruder instrument: it is a second list that has to agree with the first, and the day they disagree the answer is ambiguous. It is also immediate and legible, which is why it is there. Enabling it needs script-security 2; the config sets 3, which additionally permits passing passwords to scripts through the environment. We did not need that, and it survived because it worked. That is how permissions usually end up too wide.

The CA key has no passphrase. ovpn_initpki nopass, because provisioning is automated and a passphrase would mean a human in the loop of every apply. It is a real trade and it should be written down next to the decision rather than discovered later by whoever is reading the volume.

The last thing to say about this box is that the OpenVPN image it runs is a community one that has not seen a release in a long while. It works. It is also a dependency with no maintainer, sitting on the path into the environment, and "it works" is the argument that keeps that true for another year.

Handing out access without handing out a shell

The part that went in most recently is the least clever and has made the most difference. Nobody logs into the VPN host to mint a profile any more. A workflow does it through Systems Manager:

aws ssm send-command \
  --instance-ids "$INSTANCE_ID" \
  --document-name "AWS-RunShellScript" \
  --comment "Create VPN user" \
  --parameters commands="cd ~/ovpn && ../scripts/new_vpn_client.sh $VPN_NAME"

and the resulting profile is delivered as a Slack direct message, with the recipient resolved from their email address. The profile file is the credential, so this is not as tidy as it sounds — it now lives in a chat history we do not control, and the control is rotation rather than secrecy. Profiles are named <user>_<env>_<ddmmyyyy> so that an old one is visibly old, which is the minimum you can do when the credential is a file somebody downloaded once.

The piece I like most is the one that reads rather than writes. A second script lists the whitelist, lists the issued certificates, intersects them, and emits the result as YAML:

listCNs="sudo cat /var/lib/docker/volumes/openvpn-data-default/_data/CN_whitelist"
listCRTs="sudo ls -1 .../pki/issued/*.crt | xargs -n 1 basename"

Access becomes a file. You can diff it between weeks, review it in a pull request, and answer "who can reach production" without logging into anything. Turning a question you have to go and ask into an artefact you can read is worth more than most automation.

It is also where the two-source problem from earlier bites: an entry in one list and not the other silently disappears from the output, and an empty whitelist produces an empty file that looks exactly like a failed run. The report is only as trustworthy as the agreement between the lists it intersects.

What the complexity bought

Strip everything above and one sentence is left: no analytics requirement ever turned into a firewall exception. The vendor never had a route into the network, the database never acquired a public endpoint, and revoking the whole arrangement is docker stop.

That is the return. The costs are a tunnel pinned to a node that can be replaced underneath it, a nano instance doing two unrelated jobs, and an access-control story with one list too many in it. I would pay those again in the same order — but I would write them into the module's README on the day they were introduced rather than into a blog post two years later.