By the middle of 2024 the chain worked end to end. Someone filled in a form in Port, a GitHub Actions workflow scaffolded a repository, and Proton rendered Terraform into a pull request against our provisioning repository.

And then it stopped, because a pull request is a request, and requests are for people.

That was the whole problem. If a platform engineer reviews every rendered PR, we have not built self-service — we have built a very elaborate ticket queue with better forms. If nobody reviews them, nothing has looked at the Terraform before it reaches an AWS account. The only way out is that the review is a program, and the program's verdict is trusted enough to merge on.

What the reviewer would have looked at

Worth being precise about, because it determines what the program has to do.

These PRs are not human-written and they do not contain human mistakes. Nobody fat-fingered a CIDR. The template was reviewed when it was registered, by people, in the template repository. What is new in any given rendered PR is a combination: this template, at this version, with these inputs, in this environment, against this state.

So the reviewer is not checking style or intent. They are checking three things. Does it parse and does it reference things that exist. What would it do to the account it is pointed at. And does the template still behave the way it did when someone last read it.

Those are terraform validate, terraform plan, and a test.

The gate

The validation workflow runs on the PR, in the directory Proton rendered into, with credentials that cannot write.

name: validate
on:
  pull_request:
    paths: ['environments/**']

permissions:
  id-token: write
  contents: read
  pull-requests: write

jobs:
  validate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - id: changed
        run: |
          # Proton renders one resource per PR, so exactly one directory moves.
          dir=$(git diff --name-only origin/main... | xargs -n1 dirname \
                | sort -u | head -1)
          echo "dir=$dir" >> "$GITHUB_OUTPUT"
          echo "state_key=${dir#environments/}/terraform.tfstate" >> "$GITHUB_OUTPUT"

      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: ${{ secrets.TF_PLAN_ROLE }}   # ReadOnlyAccess + state
          aws-region: eu-west-1

      - uses: hashicorp/setup-terraform@v3
        with: { terraform_version: 1.7.5 }

      - name: Init
        working-directory: ${{ steps.changed.outputs.dir }}
        run: |
          terraform init -input=false \
            -backend-config="bucket=$TF_STATE_BUCKET" \
            -backend-config="key=${{ steps.changed.outputs.state_key }}" \
            -backend-config="region=eu-west-1"

      - name: Validate
        working-directory: ${{ steps.changed.outputs.dir }}
        run: terraform fmt -check -recursive && terraform validate

      - name: Plan
        working-directory: ${{ steps.changed.outputs.dir }}
        run: |
          terraform plan -input=false -lock=false -out=tfplan
          terraform show -json tfplan > plan.json

      - name: Refuse destructive plans
        working-directory: ${{ steps.changed.outputs.dir }}
        run: |
          deletes=$(jq '[.resource_changes[]
            | select(.change.actions | index("delete"))] | length' plan.json)
          if [ "$deletes" -gt 0 ] && [ "${DESTROY_FLAG:-false}" != "true" ]; then
            echo "::error::plan deletes $deletes resource(s) on a non-delete PR"
            exit 1
          fi

The last step is the one I would not ship this without.

A rendered PR is supposed to be additive or a bounded change. A plan that deletes things on a PR that is not a deletion means something has moved underneath us — a state key collision, a resource removed from a template between versions, someone's console click. Refusing to merge on any delete that was not explicitly asked for turned an entire category of "how did that happen" into a red check. It fires rarely. When it fires it is always right to have fired.

Two separate roles matter here as much as the checks do. The plan role is read-only plus write access to the state object; it cannot change anything in the account it is describing. The apply role, used only after the merge, can. Keeping them apart means a pull request from a source you have partly delegated to a rendering engine cannot itself become a privilege escalation.

Terratest, and where to put it

plan tells you what Terraform intends. It does not tell you whether the thing Terraform builds actually works — whether the target group ever goes healthy, whether the task role can read the parameters it needs, whether the security group lets the load balancer in.

Terratest does, because it is a Go test that really provisions.

func TestHttpServiceInstance(t *testing.T) {
	opts := &terraform.Options{
		TerraformDir: "../fixtures/http-service",
		Vars: map[string]interface{}{
			"service_instance": map[string]interface{}{
				"name":   fmt.Sprintf("tt-%s", random.UniqueId()),
				"inputs": map[string]interface{}{"port": 3000, "desired_count": 1},
			},
		},
	}
	defer terraform.Destroy(t, opts)
	terraform.InitAndApply(t, opts)

	url := terraform.Output(t, opts, "service_url")
	http_helper.HttpGetWithRetry(t, url, nil, 200, "ok", 30, 10*time.Second)
}

It is also slow and it costs money, and this is where I disagree with how we first wired it. We ran it on the rendered PR, which sounds thorough and is mostly waste: the thing under test is the template, and the template is identical across every instance rendered from it. Running it per instance means paying to re-learn the same fact, adding ten minutes to every developer's wait, and — because it stands up real infrastructure — occasionally failing for reasons that have nothing to do with the change.

Where it belongs is on the template. Register a new minor version, and Terratest runs against a fixture in a throwaway account before that version is publishable. The rendered PR then checks the combination — validate, plan, no unexpected deletes — and trusts the template, because the template earned it in its own pipeline.

We got most of the way there. Environment PRs, which are rare and expensive to get wrong, kept a Terratest run. Service instance PRs, which are frequent and identical, dropped to plan-only once the template pipeline was reliable. The median wait went from something people complained about to something they stopped mentioning, which is the only metric that ever mattered for this.

The token that stops the chain

This is the trap in the whole design, and it cost me most of a day.

The plan was: validation passes, a workflow calls gh pr merge --auto, the PR merges into main, a second workflow triggers on push to main and runs terraform apply.

The merge worked. The apply workflow never ran.

The reason is in GitHub's own documentation and is easy to read past: when you use the repository's GITHUB_TOKEN to perform tasks, the events those tasks trigger — with the exception of workflow_dispatch and repository_dispatch — will not create a new workflow run. It exists to stop a workflow recursively triggering itself, and it means that a merge performed by a workflow is, for the purposes of every other workflow, invisible.

Nothing errors. main contains the change, the PR is closed, the deployment is never applied, and Proton sits at In progress waiting for a callback from a job that was never started.

The fix is to merge as something that is not GITHUB_TOKEN. We used a GitHub App installation token — an app is auditable, its permissions are scoped, and unlike a personal access token it does not leave with whoever created it:

      - uses: actions/create-github-app-token@v1
        id: app-token
        with:
          app-id: ${{ vars.MERGE_APP_ID }}
          private-key: ${{ secrets.MERGE_APP_PRIVATE_KEY }}

      - name: Merge
        env:
          GH_TOKEN: ${{ steps.app-token.outputs.token }}
        run: gh pr merge --squash --auto "$PR_NUMBER"

--auto rather than an unconditional merge, so branch protection is still the authority. Required checks, no force pushes, and — because these PRs come from an integration rather than a colleague — a restriction on which apps may push to the branch at all. The workflow asks for the merge. Branch protection decides whether it happens. If someone deletes a required check from the ruleset by mistake, the wrong thing to have is a workflow that merges regardless.

Apply on merge

Push to main, one job, and one thing that is not optional:

concurrency:
  group: apply-${{ needs.locate.outputs.state_key }}
  cancel-in-progress: false

One apply per state key, queued rather than cancelled. Two applies against one state is how you get a lock timeout at best and a corrupted state at worst, and merges arrive in bursts because people fill in forms in bursts.

We re-planned before applying rather than reusing the plan file from the PR. That is a real trade-off and I went back and forth on it. A saved plan is the strongest guarantee Terraform offers — apply exactly what was reviewed, or refuse. But the plan was made before the merge, and between then and now another instance in the same environment may have merged and applied. A stale saved plan does not silently do the wrong thing, it fails, and then a human is back in the loop for a reason that is pure timing.

So: re-plan with -detailed-exitcode, compare the resource counts against the numbers the PR check recorded, and fail loudly if they differ. It is weaker than a saved plan and stronger than nothing, and the delete check runs again on the fresh plan.

Then, whatever happens, the callback:

      - name: Tell Proton
        if: always()
        run: |
          status=$([ "${{ job.status }}" = "success" ] && echo SUCCEEDED || echo FAILED)
          aws proton notify-resource-deployment-status-change \
            --resource-arn "$PROTON_RESOURCE_ARN" \
            --deployment-id "$PROTON_DEPLOYMENT_ID" \
            --status "$status" \
            --status-message "$(tail -c 400 apply.log)" \
            --outputs "$(terraform output -json | jq -c 'to_entries
                | map({key: .key, valueString: (.value.value | tostring)})')"

if: always(). The previous post has the story about what happens when that condition is subtler than it looks.

What it costs: merged stops meaning applied

Every branch of this design is defensible and the whole still has one property I did not like and could not remove.

In an application repository, merged means shipped, or near enough that the gap is a deploy you can watch. Here, merged means the machine agreed to try. The apply happens afterwards, in a separate run, and it can fail: a quota, an IAM boundary, an eventually-consistent dependency, a provider bug. When it does, main describes infrastructure that does not exist. The repository is now wrong, and it is wrong in the direction that is hardest to notice, because everything looks merged and green until you read the deployment status.

We handled it with alerting rather than cleverness. A failed apply pages the platform on-call, labels the merged PR, and — this is the part that mattered — sets a marker in the environment directory that makes the next PR touching that directory fail its validation. Blocking the queue behind a known-broken state is unpleasant. It is much less unpleasant than the alternative, which is a second automated change planned against a state nobody has reconciled.

I looked at merge queues for this and they solve an adjacent problem. A merge queue guarantees the checks ran against the merged result. It does not guarantee the apply after the merge succeeded, and it cannot, because the apply is not part of the merge.

Would I do it again

Yes, with three things stated up front rather than discovered.

The apply role is the only thing that can write. Everything before the merge plans with read-only credentials. The gate is a gate because the things on the wrong side of it are not capable of the thing you are gating.

Refusing is free; a bad apply is not. The delete check, the count comparison, the marker file — each of them occasionally stops a change that would have been fine, and each of them has stopped at least one that would not. When a machine is merging its own pull requests, the asymmetry is the entire safety argument.

Something must always call back. A pipeline that can finish without reporting is a pipeline that will, and the resource it forgot spends the rest of the week claiming to be mid-deployment.

What I would not do again is the part I have already admitted: running the expensive behavioural test per rendered instance rather than per template version. The rendered PR is a combination of things that were each already correct. Test the parts where they are authored, check the combination where it is assembled, and let the developer who pressed the button in Port have their service back in minutes rather than in a coffee break.