google cloud

The project quota that killed Shared VPC

The project quota that killed Shared VPC

The plan was a seven-project Google Cloud organisation. What is applied tonight is four, and the reason has nothing to do with architecture. It is a quota on a billing account, a support ticket that ran for about a month, and a decision to stop arguing and redraw the diagram.

I am writing it down because the conversation you have with yourself while collapsing a design is more useful than the design. Every layer I deleted had a reason for existing. Some of those reasons survived the deletion in another form; some turned out to have been ceremony.

What I drew first

The product is moving off a platform-as-a-service and onto Google Cloud: a containerised API, Postgres, a cache, a load balancer, three environments. The foundation question is not "which compute" — it is how many projects, and where the boundaries run, because on GCP the project is the boundary. IAM, quota, billing attribution and the blast radius of a mistake all follow project lines.

So the folder tree I wrote into bootstrap/folders.tf looked like this:

oneclub/
  platform/
    admin/                  → oneclub-admin
    networking/
      oneclub-net-dev       host project, owns the dev VPC
      oneclub-net-staging   host project, owns the staging VPC
      oneclub-net-prod      host project, owns the prod VPC
    cicd/                   → oneclub-cicd
  workloads/
    development/            → oneclub-development   (service project)
    staging/                → oneclub-staging       (service project)
    production/             → oneclub-production    (service project)

Eight projects. The shape is not exotic — it is close to what Google's own landing-zone material suggests, and the two decisions inside it are the ones worth naming.

Shared VPC with dedicated host projects. In a Shared VPC topology the network lives in a host project and the workload projects attach to it as service projects. The VPC, subnets, firewall rules, Cloud NAT and the Private Service Access range all belong to the host; Cloud SQL and Cloud Run live next door in the workload project and consume the host network through cross-project IAM. The payoff is that the people who own networking and the people who deploy services are separable — you can grant roles/compute.networkUser on a subnet without handing anyone the ability to draw new ones.

A separate CI/CD project. Its whole job was Workload Identity Federation and anything else automation-shaped: build triggers, custom build images, deployment caches, CI-only secrets. I wrote the comment defending it before I wrote the resource:

Kept separate from oneclub-admin so that "what runs CI/CD" and "what manages the platform" have independent IAM and audit trails — the kind of split that pays off the first time a CI/CD compromise needs to be scoped without losing access to platform admin.

I still think that comment is right. It is simply not something I got to have.

The number is five

I did not know there was a number. That is the honest version. I had used GCP before, inside organisations where somebody else had long since dealt with this, and I had internalised "projects are free, make as many as you need" — which is true about cost and false about quota.

New billing accounts come with a project limit, and the limit is small. Ours was five. Not five per folder, not five per environment — five projects on the billing account, total. My topology needed seven before the CI/CD project and eight with it. And one of those five was already spent: there was a plain untidy project from earlier in the year that we had been using to try things out, sitting outside the organisation entirely, quietly billing about fourteen dollars a month.

So I did what you do. I requested an increase, explained the topology, and attached the folder tree.

That request became a thread. The thread ran for roughly a month. It was not hostile and nobody ever said no in so many words — it was a sequence of clarifying questions, each of which took days, each of which I answered, with the quota unchanged at the end of every round. At some point I stopped opening it.

I want to be careful here, because "cloud support was slow" is a boring complaint and not the interesting part. The interesting part is that I let an open ticket block foundation work for a month. The whole time, the design I could actually build was available to me and I was not building it, because I was waiting for permission to build a slightly better one. That is the mistake in this story, and it is mine rather than Google's.

Collapsing seven into four

Steady state is now:

| Role | Project ID | | ----------------- | --------------------- | | Admin / control | oneclub-admin | | Workload: dev | oneclub-development | | Workload: staging | oneclub-staging | | Workload: prod | oneclub-production |

Four in the organisation, plus the untidy legacy one, which is exactly five and is the entire reason dev nearly did not survive. For a while the plan was three environments or dev — I was one project short and dev is the one you cut. Deleting the host-networking layer is what bought it back.

Three things changed shape.

The networking projects went away, and each workload project owns its own standalone VPC. Subnets, firewall, NAT and the PSA range now sit directly in oneclub-staging and oneclub-production alongside the things that use them. The module did not care: modules/vpc/ takes a single project_id, so re-pointing an environment from a host project to its workload project is a variable change.

The migration was much lighter than I had budgeted for, and for an unglamorous reason: Shared VPC had never actually been enabled. The host projects existed, the folders existed, but the environment VPCs had never been applied against them, and nothing consumed them. So what I expected to be a destroy-and-recreate became a state reconciliation — terraform state rm, then terraform import for the projects that already existed, then one apply to move them under workloads/. The lesson I will actually reuse: a topology you have not applied yet is free to change, and the window where that is true is shorter than you think.

The CI/CD project folded into admin. The WIF pool and the GitHub OIDC provider now live in oneclub-admin beside the Terraform state bucket and the Artifact Registry repository. This is a genuine loss and I am not going to pretend otherwise: the audit trail for "something deployed" and the audit trail for "something changed the platform" are now the same trail, and an identity that can do one is a short IAM hop from the other. What I have instead is narrower conditions on the federation itself — the pool's provider is bound to one GitHub organisation, and each environment's deployer service account is a separate identity that CI impersonates rather than one account that can reach every environment. It is defence at a different layer, not the same defence.

Per-environment subfolders went too. workloads/development, workloads/staging and workloads/production existed so that future per-environment projects — a data project, an AI project — would have somewhere to land. With a five-project ceiling there are no future per-environment projects, so three folders holding one project each are pure indirection. The workload projects sit directly under workloads/.

What Shared VPC was actually for

Worth asking honestly, now that it is gone.

The separation-of-duties argument is real, and it is the one I lost. At three people it is also mostly theoretical — the person who would have been granted networkUser on a subnet is the person who drew the subnet.

The argument I thought I would miss more is address-space governance: one place that owns the CIDR plan so environments cannot quietly overlap. That turned out to be solved by writing the plan down and not by owning it in a project. The allocation lives in CLAUDE.md and in three terraform.tfvars files:

  • prod 10.0.0.0/16, staging 10.1.0.0/16, dev 10.2.0.0/16
  • inside each /16: app /24, serverless-connector /28, proxy-only /24, and a /20 reserved for Private Service Access

Non-overlapping by construction, one /16 per environment with enormous room, and reviewable in a diff. A host project would have enforced this; a document plus code review also enforces it, at this size.

What I do lose concretely is centralised VPC flow-log and firewall policy — three networks to configure rather than one. Flow logs are enabled on the app subnet in all three environments because I remembered to do it three times, which is precisely the kind of thing that stops being true later.

The policies survived intact

Here is the part that reconciled me to the smaller topology. Most of what I wanted the project structure to guarantee is guaranteed by organisation policy instead, applied once at the oneclub folder and inherited by every project under it:

resource "google_org_policy_policy" "disable_sa_key_creation" {
  name   = "${local.oneclub_folder}/policies/iam.disableServiceAccountKeyCreation"
  parent = local.oneclub_folder

  spec {
    rules {
      enforce = "TRUE"
    }
  }
}

Alongside it: compute.requireShieldedVm, storage.publicAccessPrevention, sql.restrictPublicIp, a deny-all on compute.vmExternalIpAccess, and run.allowedIngress pinned to internal-and-cloud-load-balancing. Domain Restricted Sharing is enforced organisation-wide by Google's own secure-by-default posture for new organisations, so I do not declare it at all.

That first one is the one I keep coming back to. disableServiceAccountKeyCreation means no JSON key can be minted anywhere under the folder — not by me, not in a hurry, not "just for this one script". A project boundary is a wall you can walk around with enough IAM; an enforced org policy is a wall that returns an error. Losing the CI/CD project hurts less when the thing that project was mostly protecting against cannot be created in the first place.

Operator access follows the same logic. There is no VPN in this design and there will not be one: reaching a private Cloud SQL instance goes through IAP and the Cloud SQL Auth Proxy, which means every operator session is an IAM decision with a log line rather than a credential on a laptop. Combined with the external-IP deny-all, there is no VM in the organisation that can be reached from the internet at all.

Where it stands tonight

bootstrap/ is applied on the four-project model. The state bucket is 1club-tf-state with versioning and a five-generation retention rule. The workload projects are Terraform-managed google_project resources — staging and production with billing attached, development deliberately without, so it costs nothing until there is a reason for it to. Standalone VPCs are applied in staging and production; dev's is written and not applied, waiting on billing. Cloud NAT is up in staging. The WIF pool and the per-environment deployer service accounts went in this afternoon.

Cleanup is still outstanding and I am listing it because unlisted cleanup does not happen: the suspended oneclub-net-* host projects, the now-empty networking and cicd folders, the three empty workloads/* subfolders, and the auto-mode default VPCs that GCP creates in every new project and that nothing should ever use. All of that gets destroyed through Terraform — import it, mark it, apply — rather than clicked away in the console, because a resource deleted outside the state file comes back the next time somebody runs apply from a stale checkout.

Everything is Terraform 1.15.3 against the Google provider in the >= 7.32, < 8.0 band, and every leaf directory is its own root with its own state prefix in the same bucket. No compute exists yet. That is next.

What I would tell someone starting

Find the quota before you draw the diagram. Not the cost — the quota. The billing account's project limit, the per-project quotas on whatever you are about to use heavily, and the regional capacity for anything that is not an instance type you have used before. Half an hour reading limits pages would have changed my first design rather than my second.

And when a request for an exception goes past a week, build the thing you are allowed to build. The design that exists beats the design that is pending. I got a topology with a slightly bigger blast radius and a slightly weaker audit story, and I got it a month later than I needed it, because I preferred arguing about seven projects to shipping four.

Deyan Peev

Written by

Deyan Peev

Founding Engineer · Sofia, Bulgaria

Deyan Peev

Founding Engineer in Sofia, Bulgaria. Currently at 1club.

Elsewhere

© 2026 Deyan Peev