At six minutes past ten on Sunday 19 July, Sofia time, Spain and Argentina kicked off the World Cup final at MetLife Stadium. At about the same moment, I switched 1club to read-only.

For the next thirty-five minutes you could still sign in, open the schedule, see your bookings and look a member up at the front desk. You could not change anything. Behind the banner that said so, the production database left Heroku for Cloud SQL, and the API came back up on Cloud Run behind Cloudflare. When the first write landed on Google Cloud, the score was still 0-0. Nobody scored until the 106th minute.

People ask about that part, because of the final, but it went the way it did because almost nothing in it was new. This post is about the work that made the cutover boring: a staging environment first, a pipeline that only promotes what staging has already run, and two pieces of preparation the move could not do without. It is also about the fact that all of it came to roughly three weeks of my time, built with AI, in between the features, bug fixes and customer requests that did not stop for any of it.

Reliable, and not ours to configure

Heroku is famous for reliability, and nothing in this post argues with that. But it is not configurable. You get dynos, a Postgres add-on, config vars and a Procfile, and inside those lines the platform makes the decisions. For a small team that is the right trade for a long time. By this spring, two of those decisions were costing us at every release.

Every production deploy took the API down for three to five minutes. The API was one process on one Performance-M web dyno: every request, the WebSocket server and all of the scheduled jobs. A deploy meant restarting it, and until the new process was up and serving, none of that was.

Migrations also met production first. This was the entire Procfile:

release: cd server && npm run db:migrate
web: npm --prefix server start

The release phase ran every new migration straight against the production database. Apart from a developer's laptop, it was the first database those migrations had ever met, because there was no other environment. Usually that was fine. Occasionally it was not, and there is no worse place to learn that a migration does not fit the data than in front of your customers.

Even failed releases were easy to miss. A failed release does not fail the git push that triggered it: the ref updates as soon as the build succeeds, and git exits zero after printing Waiting for release.... failed. I found that out in early July, a little over two weeks before we left, while teaching our workflow to check the release through the Platform API instead of trusting the push.

In fairness, none of this is unsolvable on Heroku. It has pipelines and staging apps. But the fix we wanted was bigger than a second app. We wanted the database on a private network, a release that waits for a real health check before it takes traffic, scaling decided by concurrency, a WAF in front of the API, and deploys that authenticate by federation rather than with a long-lived HEROKU_API_KEY. That list is where Heroku's simplicity stops being a feature. If we were going to build a second environment properly, it made sense to build it somewhere we could keep building.

Staging first

The first decision made every later one cheaper: build staging before production, and make it the real thing rather than a smaller copy with different wiring.

Staging got the whole stack in its own Google Cloud project. How many projects there are, and why fewer than I wanted, is its own story. A standalone VPC. Cloud SQL for PostgreSQL 18 on a private IP. Memorystore for Valkey over Private Service Connect. Cloud Run with Direct VPC egress and Cloud NAT. A global external load balancer with a Certificate Manager certificate. Secret Manager. Workload Identity Federation, so that nobody has a service-account key. All of it is Terraform: one root per layer, thin wrappers over Google's own modules, state in one bucket.

The first VPC went in on 27 May. Three weeks later, on 16 June, GitHub Actions was deploying to staging.

Then came the change that mattered more than any single piece of infrastructure: production stopped being deployed from main before it moved anywhere. From 25 June, a production release, still to Heroku, was a manual promote of a commit that had already been through staging, behind an approval. For our last three and a half weeks on Heroku we ran the new release process against the old platform. When the cutover came, it changed where the API ran, not how we shipped it.

Staging costs a little under seventy dollars a month, most of it one container kept awake, which makes it the cheapest reliability we have bought.

One build, promoted

The rule the pipeline enforces is short: production only ever runs bytes that staging has already run, and the promotion names them.

Before any of it, a pull request passes the quality gate: formatting, lint, a typecheck of every workspace the diff touches, style guardrails, and a folder of architecture tests. Two of those tests exist because of this migration. One fails any new migration that contains destructive DDL (DROP COLUMN, SET NOT NULL, DROP INDEX, a rename) unless the file explains itself:

-- expand-contract: <why the previous revision keeps working>

The reason is that migrations now run while the previous revision is still serving every request, so a destructive one breaks the live platform in the seconds before the new code arrives. The other test fails the build if a scheduled job runs without the distributed lock. More on that below.

A merge to main deploys staging automatically:

1. Work out what changed. The API's baseline is the image live on staging and the frontends' is a git tag, so only what moved gets built.
2. Build the API image once, tagged with the commit SHA. If that tag already exists, reuse it.
3. If a migration changed, run it as a Cloud Run Job against the staging database and wait. A failed migration stops the run, and nothing after it happens.
4. Deploy the image to Cloud Run. The new revision takes no traffic until its startup probe passes, and the probe checks the database, the cache and the lock database, not just the port. Ready is not alive.
5. Deploy the changed frontends to Vercel's staging environment.
6. Only if every job above finished clean, move the staging-ready tag.

Production is a button, and nothing else deploys it. The promote reads staging-ready, never the tip of main. It waits for a reviewer, runs the production migration job if the schema moved, deploys the API image, and then deploys the changed frontends to Vercel production. A failure in either workflow posts to Slack.

The detail that makes it trustworthy is the tag. staging-ready is annotated, and the annotation records what was actually live on staging when that commit went green:

staging-ready
api-digest=sha256:…
api-sha=<commit the image was built from>
run=<staging run id>

Production deploys that digest, rather than the floating :staging tag, which a staging run finishing during the approval wait would move, or "the newest image built from an ancestor of this commit", which can pick a build whose staging deploy failed and was then reverted.

That design came out of a near miss on 17 September. A promote was dispatched forty-three seconds after a push to main. Production still resolved its commit from the tip of main at the time, so it picked the new commit almost four minutes before staging had even tagged that commit's image. Had the reviewer approved, eight frontends would have shipped the new commit while the API promoted the previous one, and which half you got would have depended on how quickly somebody clicked. Nobody approved it. The same day, production stopped reading main. I would call the pipeline bulletproof now, but mostly because of the shots it has already taken.

There is one way around the marker, for the emergency that cannot wait for a staging cycle: promote_main_tip, which ships main exactly as it was when the button was pressed, not whenever the approval lands. There is deliberately no way to promote an arbitrary commit somebody typed.

One boundary keeps the two halves apart. Terraform owns the shape of every service (scaling, probes, networking, identity), and the pipeline owns exactly two things: the image and the container environment. Terraform ignores those two fields, and the deploy step is not allowed to pass any of the others, so neither side can drift the other.

And a deploy no longer takes anything down. Cloud Run starts the new revision beside the old one, waits for readiness, moves the traffic, and sends the old instances SIGTERM. They mark themselves draining, stop their schedulers, close their WebSockets, finish what is in flight and exit, and clients reconnect. There is no step in a release at which the API is not running.

What moved, and what stayed

Only the API moved. Everything a browser loads was already on Vercel and stayed there.

The frontends (admin, the member portal, club websites, kiosk and check-in, auth, the MCP server, reports, the status page and a few more) are thirteen Vercel projects, now declared in Terraform in the infrastructure repository, one file each. Ten of them had been created by hand in the dashboard and were adopted with import blocks, with nothing recreated and no live setting changed. Almost all of them deploy to a custom staging environment on merge and to production on promote, from the same workflows as the API and always after it, so a new frontend never talks to an old API.

DNS moved weeks before the API did. Vercel had been the authoritative DNS host. Cloudflare is now authoritative for 1club.ai and 1club.me. A nameserver switch is dangerous in a way that is easy to miss: a record you forget does not fail, it just stops resolving, and you find out from whoever depended on it. So the Cloudflare zone was written in Terraform from a full export of the live Vercel zone, record by record (frontends, mail, DKIM, every verification TXT), and checked again before the switch. Frontend records are DNS-only CNAMEs to Vercel. The production API is the one proxied host, so Cloudflare's WAF and a volumetric rate limit sit in front of it, with webhooks exempt from the rate limit.

The API's edge is Cloudflare, then a global external Application Load Balancer, then Cloud Run through a serverless network endpoint group. The load balancer's certificate is validated by a DNS-authorization record rather than by traffic, so it was issued and active before a single request depended on it.

Behind it, Cloud Run keeps one warm instance and runs at most four, each taking 800 concurrent requests, with CPU always allocated for the WebSockets and the scheduled jobs. Cloud SQL for PostgreSQL 18 runs with regional high availability, on a private IP only, and holds three databases on one instance. Valkey sits behind Private Service Connect, and outbound traffic goes through Cloud NAT.

The cutover itself is a Terraform variable. The API record's shape depends on whether the load balancer's addresses are set:

api_prod_records = merge(
  var.prod_api_lb_ipv4 != null ? {} : {
    "api-prod" = { name = "api.1club.ai", type = "CNAME", content = var.origin_hostname, proxied = false, ttl = 1 }
  },
  var.prod_api_lb_ipv4 == null ? {} : {
    "api-prod-a" = { name = "api.1club.ai", type = "A", content = var.prod_api_lb_ipv4, proxied = true, ttl = 1 }
  },
  var.prod_api_lb_ipv4 == null || var.prod_api_lb_ipv6 == null ? {} : {
    "api-prod-aaaa" = { name = "api.1club.ai", type = "AAAA", content = var.prod_api_lb_ipv6, proxied = true, ttl = 1 }
  },
)

Unset, it is a DNS-only CNAME to Heroku. Set, it becomes proxied A and AAAA records at the load balancer. They are proxied because Cloudflare's protection only applies to traffic that passes through it. Cutting over means uncommenting two lines in a .tfvars file and applying. Rolling back means commenting them out again.

The plan that changed on 9 July

The first plan for the data was Google's Database Migration Service: continuous replication from Heroku Postgres into Cloud SQL, a promotion at cutover, and no freeze at all. It needs logical replication switched on at the source. On Heroku that meant checking what our Postgres plan allowed, possibly upgrading it for the window, and then running a replication job across two providers for days. On 9 July I replaced all of that with something much less clever: pg_dump to a plain-SQL file, a bucket, and gcloud sql import. The database is small enough that the dump and the import take minutes.

Replication buys you no freeze. A dump is a copy of one moment, so for as long as the copy takes, nothing may change. Accepting a freeze was easy. Deciding what it should look like to the people using the platform took more thought.

The same week, the whole production stack was applied and rehearsed with a throwaway copy of the data imported into it. Then, because the cutover was ten days away, I switched production off. Cloud SQL was stopped, Cloud Run scaled to zero, and the Valkey node and the load balancer destroyed, with each step and its reverse written into the runbook. For ten days production existed, had been tested, and cost next to nothing.

The preparation

The extra database: keeping the jobs in sync

On Heroku, every scheduled job ran inside the one web dyno, with node-cron, in process. At the time of the cutover there were twenty-six named jobs, among them recurring transactions, subscription reconciliation, trial expiry, booking reminders, notification and domain-event processing, end-of-day booking status, waitlist expiry and webhook delivery, plus a bill run for each billing schedule. One process meant each job ran once, by construction.

Cloud Run takes that guarantee away. Production runs at least one instance and up to four, and during every deploy the old revision's instances are still alive while the new ones start. Every instance runs the same scheduler. Without coordination, every job runs on every instance: a bill run charges twice, and a reminder is sent twice.

So a third database sits on the Cloud SQL instance, next to the main one and the analytics one. oneclub_locks holds one table, and a scheduled job has to win a row in it before it runs:

INSERT INTO cron_locks ("jobName", "lockedBy", "acquiredAt", "expiresAt", "heartbeatAt", "createdAt", "updatedAt")
VALUES ($1, $2, now(), now() + make_interval(secs => $3), now(), now(), now())
ON CONFLICT ("jobName") DO UPDATE SET
  "lockedBy"    = EXCLUDED."lockedBy",
  "acquiredAt"  = now(),
  "expiresAt"   = EXCLUDED."expiresAt",
  "heartbeatAt" = now(),
  "updatedAt"   = now()
WHERE cron_locks."expiresAt" < now()
RETURNING "lockedBy"

It is one atomic statement: insert the lease, or take it over only if it has expired. The instance that gets its own id back runs the tick, and every other instance skips it. A lease lasts five minutes and a heartbeat extends it while the job works, so a crashed holder's lease simply runs out and a later tick reclaims it. At shutdown, an instance releases only the leases it is not using. A lease held by work that is still running is left to expire, so a peer cannot start a job that is still executing somewhere else.

Three decisions in it matter more than the SQL.

It fails closed. If the lock database cannot be reached, acquisition retries a couple of times, then the tick is skipped and logged as an error. A skipped reminder is recoverable. A double charge is a support ticket and a refund.

It is part of readiness. A revision whose lock database does not answer never takes traffic, because an API that serves requests perfectly while nothing scheduled runs is exactly the kind of half-alive state that goes unnoticed for days.

It is enforced by CI. An architecture test counts the scheduling calls in every job file and fails the build if any of them runs outside the lock. It exists because the rule kept leaking: an in-process isProcessing flag looks like a guard and does nothing across containers. There were twenty-six jobs at the cutover. There are forty-five now, and nobody has had to remember the rule for any of them.

Keeping it in its own database, rather than as a table in the main one, has a practical upside: the whole feature is switched on by the presence of one connection string. Locally there is no lock URL and the jobs run directly. In the cloud the URL is mounted and they coordinate. It has its own tiny schema and migrations, and it never touches a business table. Its pool is small, but it still counts against the same server-wide connection limit as everything else, which is a budget of its own.

It went in on 15 June, on staging, where it could be wrong safely, and the last jobs moved under it on 7 and 8 July. By the cutover it had been deciding who runs what for a month.

The switch: read-only, not down

Heroku has a maintenance mode, and the runbook's first draft used it. Turn it on and every request gets an error page, which is an outage with a nicer message.

What I wanted was a platform that stays useful while it cannot change. So on the morning of 9 July, the same day the runbook switched to a dump, READONLY_MODE became a platform-wide switch. It went in as one commit across the server, both admin apps, the member app, the kiosk and the public sites. The server half is one middleware, shown here without the part that echoes the origin for CORS:

export function readonlyModeGuard(req: Request, res: Response, next: NextFunction): void {
  if (!getReadonlyMode()) return next();
  if (!MUTATING_METHODS.has(req.method) && !OAUTH_SIGNIN_PATH.test(req.path)) return next();
  if (MUTATING_METHODS.has(req.method) && EXEMPT_PATHS.has(req.path)) return next();

  res.status(503).header('Retry-After', '3600').json({
    error: 'The platform is temporarily in read-only mode. Changes are disabled - please try again later.',
    code: 'READONLY_MODE',
  });
}

Reads pass, and writes get a 503. Every POST, PUT, PATCH and DELETE is refused, with a Retry-After and a machine-readable code.

Signing in still works. Login, logout, token refresh and switching organisation or club are exempt. They write session rows and nothing else, and a read-only platform nobody can sign into is no better than an outage. Signup is not exempt, because it creates records.

OAuth sign-in is blocked, even on GET. A first-time Google, Facebook or Apple sign-in creates a user, and the Google and Facebook callbacks arrive as GET requests, which the method check would wave through. The first version exempted OAuth callbacks like login. Nine minutes later, a second commit blocked them, because a user created on Heroku during the freeze would be a user who never reached Google Cloud.

Webhooks are refused on purpose. The guard is registered before the raw-body webhook routes, so Stripe, myPOS and SendGrid get a 503 too, and they retry. Their events replay once writes are back, on the new platform, instead of being written into a database that is about to be left behind.

Scheduled jobs stand still. Because every job goes through the lock, one check at the top of withCronLock silences all of them. The bill-run scheduler, the one thing that scheduled itself directly, got a check of its own.

The frontends say so. Each app reads the flag from an unauthenticated GET /v1/config, so it works on the login page too, and shows a translated banner. A blocked change fails visibly, with a toast that says why, instead of a dialog closing as if the change had been saved.

The difference from a maintenance page is who notices. For thirty-five minutes on a Sunday night, a member could still open the app and see their bookings, and staff could still look things up. The only thing nobody could do was change something, and the banner told them so before they tried.

The day

I spent the whole day of the final preparing. It started at lunchtime, with production coming back from its pause layer by layer: Cloud SQL started, Cloud Run back to one warm instance, the Valkey node recreated, and the organisation-policy exception that lets the load balancer invoke a public Cloud Run service. At 19:43 the API certificate's DNS-authorization record went into Cloudflare, so the certificate could be issued well before anything needed it. By the evening, everything the window needed was in place, and there was a rollback at every point. Until Heroku was switched off, going back meant re-commenting two lines and lifting the read-only flag. Heroku's data would still be exactly right, because it had been read-only the whole time.

Thirty-five minutes in the first half

The window itself was seven steps.

1. Freeze. READONLY_MODE on, on Heroku. The banner went up everywhere at once.

2. Dump. pg_dump from Heroku as plain SQL, because gcloud sql import cannot read the custom format. That meant the main database and the analytics one. The lock database had nothing to bring, because it had never existed on Heroku.

3. Import. The files go to a bucket and Cloud SQL imports them itself, server-side, because the instance has no public IP and an organisation policy forbids giving it one. Before the data, one line runs as postgres:

   GRANT cloudsqlsuperuser TO oneclub_app;
   

   That lets the dump be imported as the application user, so every object in it, extensions included, is owned by the role that future migrations run as. Get that wrong and the next prisma migrate deploy cannot alter its own tables.

4. Wire. A script composes the connection-string secrets from Terraform outputs and the seeded password.

5. Deploy. The current image, through the migration job, which had nothing to do: the migration history came across inside the dump.

6. Flip. Two lines uncommented in the Cloudflare .tfvars, then terraform apply, and api.1club.ai became proxied A and AAAA records at the load balancer.

7. Verify. Health checks, a real write, a lease row appearing in oneclub_locks, and a WebSocket connecting through the public hostname.

Google Cloud never had READONLY_MODE set, so writes came back the moment requests arrived there. It took thirty-five minutes, start to finish, inside the first half. Ferran Torres scored in the 106th minute, by which time production had been taking writes on Google Cloud for more than an hour.

After the final whistle I fixed a Book button that offered classes the API would then refuse. I rewrote the production workflow to promote the staging image to Cloud Run instead of pushing to Heroku, and moved the deploy steps both workflows share into composite actions. The last commit went in at five past two. The next morning I was back on the new check-in app.

Heroku stayed as the rollback for another three and a half weeks. Its configuration left the repository on 13 August.

Two months on

That was twelve weeks ago. Since then, the platform has not had a single cloud-side incident. Releases, which used to cost three to five minutes of downtime each, now cost none, and a migration meets staging before it meets production.

What the two months did contain was tuning, and most of it is in the history:

• Health checks split in two. A deep endpoint gates startup, and a shallow one with no I/O decides restarts, so a database blip cannot become a restart storm.
• Concurrency went up while the instance count stayed down. The limit is 800 concurrent requests per instance and at most four instances, because the real constraint is database connections, not CPU: every instance divides the same max_connections between its pools.
• A near miss in Terraform. max_connections had been raised to 200 on the production instance outside Terraform. The module manages database flags as one list, and our list was empty, so the next apply for any unrelated reason would have removed the setting. That would have dropped the limit back to 100 while the API was sized for about 190. The setting is declared now, and the plan is empty.
• Old revisions actually leave. An open WebSocket counts as a request in flight, and Cloud Run keeps an instance while it holds one, so the request timeout came down to fifteen minutes to let old revisions drain. The latency alert also learned to ignore WebSocket upgrades.
• Monitoring that knows what matters. Uptime checks from three continents need two regions to agree before anything alerts. Resource alerts cover Cloud Run and Cloud SQL, everything lands in a Slack channel, and a status page at status.1club.ai reads Cloud Monitoring through federation, so it holds no key either.

And one correction. In May I wrote that there would be no VPN in this design, and that operators would reach the private database through IAP and the Cloud SQL Auth Proxy. There is a VPN now: a small WireGuard server per environment, because per-port tunnels turned out to be miserable for an ad-hoc production query or a bit of surgery on cache keys. Tailscale and Cloudflare WARP were both considered and rejected, to keep a vendor out of the path to production data. The earlier post's security reasoning still holds, but it got the ergonomics wrong.

Three weeks, with AI

Counted in days I actually spent on it, all of this comes to about three weeks: the Terraform, the pipeline, the lock, the read-only mode, the runbooks and the night itself. On the calendar it ran from late May to the final, because none of those days were spent only on this. Of the 424 commits I made to the application repository in those eight weeks, about sixty touched deployment or infrastructure. The rest were product work: a new self-service check-in app, billing and promotion fixes, customer requests and the usual bugs.

I built it with AI, mostly Claude Code, and the division of labour was simple. The agent wrote code, read documentation and drafted runbooks. I made the decisions, reviewed every change and ran every apply. The infrastructure repository's instructions to the agent say so in as many words:

Do not run `terraform apply` (or `terraform plan -out`, `terraform destroy`, or any
state-mutating command). The user applies all changes themselves.

Every session that touches Terraform ends with a list of directories to apply, in dependency order, and nothing reaches Google Cloud until a person has read the plan. That rule is a large part of why I trust everything else.

AI mostly lowered the cost of the careful parts. The read-only mode, across the server and seven apps, translated and tested, was one commit and two small follow-ups. The runbook has a rollback at every step. The import was rehearsed. Production was built, tested and switched off for ten days to save money. The lock has a CI test so that nobody can forget it. Each of those is the thing you skip when a migration is a side project squeezed between customer requests, and the thing whose absence you discover late at night with a dump half-imported. None of it had to be skipped, and that is why the night was boring.