The API is an Express process on Cloud Run with forty-odd scheduled jobs and a WebSocket server living inside the same container. Getting its health probes right took three passes over four weeks, and each pass was wrong in a way the previous one could not have shown me.

The endpoints are now /v1/health/ready and /v1/health/live, they answer different questions, and one of them does no I/O at all. Here is how that happened.

Pass one: the default is a port check

If you deploy a Cloud Run service with no startup probe, Cloud Run decides the revision is ready when something accepts a TCP connection on the container port. That is the whole test.

For an Express app, app.listen() happens almost immediately — before the database pool has connected, before Redis is reachable, before the process has established that any of its configuration is valid. So a revision whose DATABASE_URL points at a private IP it cannot route to will bind the port, pass the check, be marked ready, receive one hundred percent of traffic and serve errors. The deploy reports success. Nothing in the pipeline objects.

The first fix was a startup probe pointed at the app's existing deep health endpoint:

startup_probe {
  http_get {
    path = var.startup_healthcheck_path
    port = var.container_port
  }
  initial_delay_seconds = 0
  period_seconds        = 10
  timeout_seconds       = 10
  failure_threshold     = 24
}

Three numbers worth defending. timeout_seconds = 10 because Redis's client uses a ten-second connect timeout, and a cold first connection can use most of it — a five-second probe timeout turns a slow-but-fine dependency into a failed deploy. period_seconds × failure_threshold = 240s, which is Cloud Run's maximum startup window; going higher is not permitted, and going much lower means a container that is merely slow gets killed and retried, which is how you turn a ten-second delay into a five-minute rollout.

This was a real improvement. A revision that cannot reach its dependencies now fails its deploy instead of serving.

Pass two: coupling liveness to the same endpoint

Having built a good deep check, the obvious next move was to also use it as the liveness probe. If the container is unhealthy, restart it. That is what liveness probes are for.

It is not what liveness probes are for, and the difference is the thing this whole post exists to say.

Readiness answers: may this instance receive traffic? A negative answer takes one instance out of rotation. It is cheap, reversible, and local.

Liveness answers: should this container be destroyed and replaced? A negative answer kills the process.

Now imagine the database is unreachable for four minutes — a failover, a maintenance window, a network blip on the PSA path. Every instance's deep check fails. Every instance's liveness probe fails. Cloud Run kills all of them, starts fresh containers, and those fresh containers run a startup probe against the same unreachable database, fail it, and get killed too. You have converted a dependency outage into a restart storm, thrown away every warm connection pool and every in-flight request, and the database — which is the thing that is actually struggling — is now being hammered by a fleet of cold processes all opening new connections at once.

The restart fixed nothing, because the problem was never inside the container.

This is the rule I would put on a wall: a liveness probe should only test things a restart can repair. A wedged event loop, a deadlock, an unrecoverable internal state. Never a dependency. The moment a liveness check reaches across the network, it has become a mechanism for amplifying other people's outages.

Pass three: two endpoints, two contracts

So they split. /v1/health/ready is the startup-probe target and gates deeply:

const [database, redis, lockDatabase, analyticsDatabase] = await Promise.all([
  checkMainDatabase(),
  checkRedis(),
  checkLockDatabase(),
  checkAnalyticsDatabase(),
]);

const gatingUnhealthy =
  database.status === 'unhealthy' ||
  redis.status === 'unhealthy' ||
  lockDatabase.status === 'unhealthy';

Three dependencies gate, one does not, and the asymmetry is the interesting part.

Redis gates because without it sessions silently fall back to per-instance memory. That is not an outage — it is worse than an outage, because everything appears to work while users get logged out whenever they land on a different instance. A failure mode that looks like success should never be allowed to pass a readiness check.

The lock database gates because the scheduled jobs use it to decide which instance runs a given tick. Without it, every background job on the platform fails closed. An API that answers requests perfectly while nothing scheduled runs is exactly the sort of half-alive state that goes unnoticed for days.

Analytics deliberately does not gate. It powers admin dashboards. If the analytics database is down and analytics is a gating dependency, then a reporting outage removes core API capacity — bookings stop, check-in stops, payments stop — to protect a chart. The endpoint reports degraded and returns 200. Shipping broken dashboards is the better failure.

/v1/health/live is the other half, and its implementation is the shortest function in the server:

app.get('/v1/health/live', (req, res) => {
  if (isDraining()) {
    return res.status(503).json({ status: 'draining', timestamp: new Date().toISOString() });
  }
  res.json({ status: 'ok', timestamp: new Date().toISOString() });
});

No I/O. It proves the event loop is turning and the process has not begun shutting down. That is the entire contract, and its probe settings match: period_seconds = 30, timeout_seconds = 5, failure_threshold = 3 — ninety seconds of sustained unresponsiveness before a restart. The timeout can be tight precisely because the handler does nothing; if it cannot answer in five seconds, the event loop is genuinely blocked, which is a thing a restart repairs.

The original /v1/health stayed, unchanged. It is the Dockerfile's HEALTHCHECK target for local Docker and there was no reason to break it. Endpoint churn during a probe migration is how you end up with a revision whose probe points at a path the image does not serve yet.

Startup is not continuous readiness

One honest limitation, because it took me a while to internalise it: Cloud Run's startup probe runs at startup. It gates whether a new revision or a scale-up instance may begin serving. It does not run continuously, so an instance that was healthy at boot and has since lost its database will not be pulled out of rotation — it will keep receiving traffic and keep failing requests.

Kubernetes has a separate continuous readiness probe for this. Cloud Run does not. What it has instead is liveness, which — as established above — is the wrong tool. So the gap is real and the mitigation lives elsewhere: in the application returning honest errors, and in monitoring that watches the error rate rather than watching a probe.

I would rather write that down than let the probe configuration imply a guarantee it does not make.

Draining is a third answer

All three endpoints share one early return:

if (isDraining()) {
  return res.status(503).json({ status: 'draining', ... });
}

On SIGTERM the process marks itself draining before it does anything else, and from that moment every health endpoint reports 503. Then it stops the schedulers, closes WebSocket connections, gives in-flight HTTP up to four seconds to finish, releases the job leases it is not actively using, flushes telemetry and closes its stores. A watchdog forces exit at nine seconds regardless.

The ordering is the point. Marking draining first means the platform's own health checks stop claiming this instance is available before the instance starts dismantling itself, rather than in the middle of it. And leases held by work still running are deliberately not released — they expire on their TTL instead, so another instance cannot pick up a job that is still executing here. A clean shutdown that hands your half-finished work to a peer is not a clean shutdown.

Concurrency is a probe question wearing a different hat

The last change this week was request concurrency: 800 per instance in production, with a maximum of four instances.

Those numbers look like scaling knobs and they are really a connection budget. The server computes its database pool size from the real max_connections of the instance divided by the peak number of application instances — both passed in as deployed configuration rather than guessed — and refuses to start if either is missing. So max_instances is not "how much traffic can we take", it is "how many processes are allowed to exist at once, each holding a slice of a fixed number of database connections."

Raising concurrency rather than instance count is the cheap direction: one process serving eight hundred concurrent requests holds one pool. Eight processes serving one hundred each hold eight. For a workload that is mostly waiting on I/O — which this is — the CPU cost of the extra concurrency is small and the connection cost of the extra instances is not.

There is an interaction with the probes worth flagging. Long-lived WebSocket connections each count as one concurrent request, and Cloud Run will not reclaim an instance holding an open request. So when a new revision takes over, the old one keeps every instance that still has a socket attached, and both revisions bill simultaneously until those sockets are severed. Readiness says nothing about this; the instance is perfectly healthy. It is just not going away.

What a passing probe still does not tell you

/v1/health/ready proves the dependencies are reachable from this container. It does not prove the container did everything it was supposed to do at startup.

Some of those forty scheduled jobs catch and log their own initialisation failures rather than crashing the process — which is the right call for most of them, because one broken poller should not take the API down. The consequence is that a revision can pass a deep readiness check, serve traffic correctly, and still have quietly failed to schedule something.

A healthy HTTP listener certifies the HTTP listener. It is not a statement about the work happening beside it, and I have started to think that gap — between "the service answers" and "the service is doing its job" — is where most of the uncomfortable production surprises actually live.