On the fifteenth of June I changed one boolean in a terraform.tfvars and took staging's largest line item from about fifty-eight dollars a month to about thirteen. I wrote a pleased little comment above it:

# Cost tuning — staging only. Set explicitly here (not in the shared module) so
# prod stays decoupled and picks its own warmth/cost trade-off in its own tfvars.
# cpu_idle=true throttles CPU between requests: we keep one warm instance
# (min=1, no cold starts) but only pay full CPU while serving, not 24/7. This is
# the main lever on the always-on bill. 1Gi is enough for staging.
cpu_idle     = true
memory_limit = "1Gi"

Every sentence in that comment is true. It is still the wrong configuration, and on Monday I am putting the forty-five dollars back, because in the ten weeks since I wrote it not one scheduled job has run in that environment.

What cpu_idle actually does

Cloud Run has two billing models and the container's CPU allocation chooses between them.

With cpu_idle = true — request-based billing — the instance is allocated CPU while it is handling a request and throttled to approximately nothing when it is not. Keeping min_instances = 1 means a container stays resident, so there are no cold starts, but between requests that container is not running. Its memory is intact, its sockets are open, its timers are armed. It simply does not get scheduled onto a core.

With cpu_idle = false — instance-based billing — CPU is allocated for the entire lifetime of the instance, and you are billed for that lifetime whether or not anything arrives.

If your service is a request/response API, the first option is close to free money. If your service has anything living between requests, the first option silently deletes it.

Forty scheduled jobs living between requests

The API is a single Express process, and it carries its own scheduler. There are around forty jobs in server/jobs/ and they all look roughly like this:

export class SessionSweepJob {
  private readonly CRON_SCHEDULE = '15 4 * * *'; // Daily at 4:15 AM UTC

  start(): void {
    this.task = cron.schedule(this.CRON_SCHEDULE, async () => {
      await withCronLock(CRON_JOB.SESSION_SWEEP, () => this.sweepOnce());
    });
    logger.info({ schedule: this.CRON_SCHEDULE }, '[SessionSweep] Scheduled');
  }
}

That is node-cron. Which is to say: it is a JavaScript timer. It fires by the event loop coming around and finding that the moment has arrived.

An event loop requires CPU to come around.

So a daily job scheduled for 04:15 UTC, in an environment where nobody sends a request at 04:15 UTC, on an instance whose CPU is throttled to zero between requests, does not run. Not late — not at all. The next time a request arrives and the container is given CPU again, the timer's moment has long passed; node-cron computes the next occurrence and waits for that one, which it will also miss for exactly the same reason.

The jobs that do occasionally fire are the ones whose schedule happens to land while someone is clicking around in staging. Which means the behaviour is not "broken", it is intermittent and correlated with whether a human is looking — close to the worst diagnostic signature a bug can have.

Nothing told me

This is the part I want to be precise about, because "I saved money and broke something" is only half the story. The other half is that every instrument I had reported success.

• The deploy succeeded. The image is fine.
• The startup probe passed. It makes a request, which allocates CPU, which lets the readiness handler reach the database and Redis and answer 200.
• The liveness probe passed, for the same reason.
• The uptime checks I set up earlier this month passed. They are requests.
• The bill went down, which is the outcome I had asked for.
• The jobs log start, finish and skip-overlap — so there is a record of every run. There is no record of a non-run, because not happening does not emit a log line.

Every probe in the system is request-shaped, and the failure was specifically a failure of the part that does not involve requests. I had built a monitoring surface that could not see the thing I broke, and the fact that the broken thing was invisible is precisely why the cost saving looked so clean.

I found it the ordinary way in the end: I went looking for a record of a job I knew should have run, and there wasn't one, and then there wasn't one for anything else either.

Putting it back

cpu_idle     = false
memory_limit = "2Gi"

And the arithmetic, which now lives in PRICING.md so nobody has to rediscover it:

• CPU: 1 vCPU × 730 h @ $0.000018/vCPU-s → ~$47.30
• Memory: 2 GiB × 730 h @ $0.000002/GiB-s → ~$10.51
• Requests: @ $0.40/million, at staging volume → < $0.10

About fifty-eight dollars. The previous shape billed its idle minimum instance at the min-instance rates instead — $0.0000025 per vCPU-second and per GiB-second — which came to roughly thirteen. So the change is about forty-five dollars a month, and it makes Cloud Run staging's biggest line by a wide margin: five to six times the database sitting next to it, which is a db-f1-micro costing around ten.

Paying six times as much for the process as for its Postgres instance looks absurd on a spreadsheet. It stops looking absurd the moment you write down what the alternative was actually delivering.

The options I did not take

Move the jobs to Cloud Scheduler. Genuinely cheaper — the first three jobs are free and the rest are cents each. It also means forty authenticated HTTP endpoints, forty schedule definitions to keep in step with the code that implements them, and every job newly bounded by a request timeout rather than by its own runtime. Most of all it means staging's execution model would no longer match production's, and the entire purpose of staging is that it matches production. I am not willing to buy forty-five dollars by making the rehearsal environment rehearse something else.

Run a second, worker-shaped service. That service would need always-allocated CPU, so I would be paying the fifty-eight dollars over there instead — plus a second deployment, a second image rollout, and a new class of version skew between the API and the jobs that share its code.

Drop to min_instances = 0. Then there is no container at all between requests, so the jobs do not run and cold starts arrive as a bonus.

The honest conclusion is that an in-process scheduler and request-based billing are incompatible, and the fix for that is either to stop using an in-process scheduler or to pay for the CPU. Given that the scheduler is shared with production and works well there, paying is the cheaper of the two.

The hazard waiting on the other side

Production has had cpu_idle = false since the day it was scaffolded — I had written the reason into its tfvars before staging ever made the mistake:

# cpu_idle=false → CPU always allocated: required for WebSocket keepalives and
# the in-process node-cron scheduler. NOTE: with min>=2, node-cron runs on every
# instance — duplicate execution is prevented ONLY by the distributed lock in
# the oneclub_locks DB, so USC1_LOCK_DATABASE_URL MUST be seeded + wired.

That note matters because the two problems are mirror images. Throttle the CPU and a job runs zero times. Scale to four instances and it runs four times, since every container has its own scheduler and all of them wake at 04:15.

The dedupe is a lease table in a small separate database. Each job takes a row keyed by its own name before doing anything; whoever wins the row runs, the rest skip that tick. The lease has a five-minute TTL — longer than any job's runtime — extended by a heartbeat while the handler is working, so a container that dies mid-job does not hold the key forever. Ownership is checked on renew and release against an identity built from Cloud Run's revision name, the process id and a random suffix, so an instance can only ever release its own hold.

It is also why the lock database is one of the dependencies that gates readiness: if it is configured and unreachable, every background job on the platform fails closed, and an instance in that state should not be taking traffic and pretending otherwise.

None of which applies to staging, where max_instances is pinned at one. That is what makes Monday's change safe to apply on its own: the environment cannot produce a second scheduler to conflict with the first.

What the forty-five dollars actually bought

Not warmth. The instance was already warm — min_instances = 1 had been true the whole time, and there were no cold starts under either configuration.

What it bought was continuity: the container's right to keep executing when nobody is asking it for anything. For a stateless API that is worth nothing. For a process that holds WebSocket keepalives and forty timers, it is the difference between a service and a very responsive corpse.

The lesson I am taking is narrower than "cloud costs are subtle". It is this: a cost optimisation that changes the execution model is not a cost optimisation. It is a behaviour change with a discount attached, and it should be reviewed as a behaviour change — by asking what runs between requests — rather than waved through because the graph went down and the probes stayed green.

Staging got cheaper in June because it had quietly stopped doing part of its job, and every measurement I had was pointed at the part it was still doing.