The agent went online a fortnight ago. The first thing I put on it was not a build.

The builds were never the interesting failures. The interesting failures were "somebody changed a box" — an execution policy relaxed for one debugging session and left there, a Defender exclusion that a group policy refresh quietly reverted, a code-signing certificate that nobody was watching. Every one of those shows up as a red build eventually, at the worst possible moment, looking like a code problem.

So: a scheduled job that runs on the Windows agents every weeknight, collects what the machines actually look like, compares that against a baseline in the repository, and tells somebody when the two disagree.

The cron line

triggers {
  cron('TZ=Europe/Sofia\nH 2 * * 1-5')
}

Three things in one line.

H is not "hourly". It is a hash of the job's name into the allowed range. 0 2 * * * across nine jobs means nine jobs starting at exactly 02:00:00 and a controller that falls over on the load spike. H 2 gives this job some minute between 02:00 and 02:59, and gives it the same minute every time, so the schedule is still deterministic and you can still reason about it. Use H in every field you do not have a specific reason to pin.

The timezone is the controller's, not the agent's. Which is fine until your controller is in UTC and your operations people are not, and "the overnight job" starts arriving at 04:00 local in summer. The TZ= line at the top of the spec fixes it explicitly, and explicit is worth two lines of documentation you will not write.

1-5 because nobody reads it at the weekend. A drift report that lands on Saturday morning is a drift report that gets marked read on Monday.

The skeleton

pipeline {
  agent { label 'windows' }

  options {
    timestamps()
    skipDefaultCheckout()
    disableConcurrentBuilds()
    timeout(time: 20, unit: 'MINUTES')
    buildDiscarder(logRotator(numToKeepStr: '60'))
  }

  triggers {
    cron('TZ=Europe/Sofia\nH 2 * * 1-5')
  }

  parameters {
    booleanParam(name: 'FAIL_ON_DRIFT', defaultValue: true,
                 description: 'Uncheck for a dry run against a new baseline')
  }

  stages {
    stage('Checkout') {
      steps {
        checkout scm
      }
    }
    stage('Collect') {
      steps {
        powershell script: 'ci\\collect.ps1 -OutFile facts.json', label: 'collect facts'
      }
    }
    stage('Compare') {
      steps {
        script {
          def facts    = readJSON file: 'facts.json'
          def baseline = readJSON file: "ci/baseline/${env.NODE_NAME}.json"
          compare(facts, baseline)
        }
      }
    }
  }

  post {
    always  { archiveArtifacts artifacts: 'facts.json,drift.json',
                               allowEmptyArchive: true, fingerprint: true }
    unstable { notify('drifted') }
    failure  { notify('failed to run') }
    fixed    { notify('back to baseline') }
  }
}

skipDefaultCheckout() and then an explicit checkout scm looks redundant and is not. The default checkout happens before options has finished being useful, and I want the checkout to be a stage I can see the timing of. timeout is there because a hung Get-ChildItem Cert:\ will otherwise sit on your only Windows agent until somebody notices in the morning.

disableConcurrentBuilds() because two of these racing on the same machine will produce two different answers and you will believe the wrong one.

The step that does not fail

This is the paragraph I would keep if I could only keep one.

powershell 'Get-Item C:\\does-not-exist'

That is green. Get-Item raises a non-terminating error, PowerShell writes it to the error stream, execution continues, the script ends, and the process exits zero. Durable Task sees zero and reports success. Your nightly check has been passing for three weeks and checking nothing.

Depending on which version of the PowerShell plugin you have, the step may set $ErrorActionPreference for you. Do not rely on it. Set it yourself, on the first line of every script, along with strict mode:

$ErrorActionPreference = 'Stop'
Set-StrictMode -Version Latest

That covers cmdlets. It does not cover native executables, which do not raise PowerShell errors at all — they set $LASTEXITCODE and carry on. So there is a wrapper, and everything goes through it:

function Invoke-Native {
  [CmdletBinding()]
  param(
    [Parameter(Mandatory)][string]   $Exe,
    [Parameter(ValueFromRemainingArguments)][string[]] $Arguments = @()
  )
  & $Exe @Arguments
  if ($LASTEXITCODE -ne 0) {
    throw "$Exe exited with $LASTEXITCODE"
  }
}

And when you genuinely want the exit code rather than a failure, ask Jenkins for it instead of inventing something:

def status = powershell(returnStatus: true, script: 'ci\\probe.ps1')
if (status == 2) {
  unstable('probe reported degraded')
}

The interpolation bug

I lost an evening to this one and I have watched two other people lose one since.

powershell """
  Write-Host "running on $env:COMPUTERNAME"
"""

Groovy sees a double-quoted string, finds $env, and interpolates it — because env is a real object in a pipeline. What reaches PowerShell is the toString() of an EnvActionImpl followed by a literal :COMPUTERNAME. If you are luckier, you write $_ somewhere and get a MissingPropertyException for a property called _, which at least fails loudly.

The rule, with no exceptions: PowerShell lives in single-quoted Groovy strings. ''' for anything multi-line. If you need a Jenkins value on the inside, it goes through the environment, not through the string:

withEnv(["BASELINE=ci/baseline/${env.NODE_NAME}.json"]) {
  powershell '''
    $ErrorActionPreference = 'Stop'
    $baseline = Get-Content -Raw -Path $env:BASELINE | ConvertFrom-Json
  '''
}

This is also the security answer. An interpolated parameter is string concatenation into a shell, which is the oldest injection in the book, and a build parameter is attacker-controlled the moment anyone outside your team can trigger the job.

Credentials that stay out of the log

withCredentials([usernamePassword(credentialsId: 'drift-check-reader',
                                  usernameVariable: 'CHECK_USER',
                                  passwordVariable: 'CHECK_PASS')]) {
  powershell '''
    $ErrorActionPreference = 'Stop'
    $secure = ConvertTo-SecureString $env:CHECK_PASS -AsPlainText -Force
    $cred   = [System.Management.Automation.PSCredential]::new($env:CHECK_USER, $secure)
    Invoke-Command -ComputerName $env:TARGET -Credential $cred -ScriptBlock { ... }
  '''
}

The masking is exact-match. Jenkins replaces the literal secret in the console output; it cannot recognise the same secret after you have base64-encoded it, URL-escaped it, or interpolated it into a connection string. So the rule about never echoing a secret is not a rule about the masker being weak, it is a rule about the masker only being able to do the obvious case.

One Windows-specific hazard: if PowerShell script block logging is enabled by policy on the agent, the body of every script block goes into the event log, literals included. That is a good control on a workstation and a secret disclosure on a build agent. Decide which one this machine is.

Handing data back to Groovy

There are two ways and only one of them survives contact.

The first is returnStdout: true and parsing the string. It works for a single value. It also captures everything anyone Write-Hosted, every progress bar, and whatever a module decided to print on import, so as soon as the output is structured you are writing a parser to strip other people's chatter out of your data.

The second is a file:

$facts | ConvertTo-Json -Depth 8 | Set-Content -Path $OutFile -Encoding utf8

def facts = readJSON file: 'facts.json'

Two arguments in there are load-bearing.

-Depth 8. ConvertTo-Json defaults to a depth of two. Anything below that is silently replaced by the type name as a string. Not an error, not a warning — your nested object becomes "System.Object[]" and your comparison starts reporting that nothing ever changes. I have been caught by this twice.

-Encoding utf8. On Windows PowerShell 5.1 that writes a BOM. readJSON copes; plenty of other things do not, and a BOM in front of a { is a famously baffling parse error. On pwsh 7, utf8 is BOM-less and utf8BOM is the opt-in, which is the right way round and one more argument for doing file-producing work in 7.

While you are at it, wrap every collection in @() before it goes into the JSON. PowerShell unrolls a one-element array into the element, so a machine with a single certificate serialises an object where every other machine serialises a list, and your comparison code gets to find out at runtime.

Drift is unstable, not failed

def compare(facts, baseline) {
  def findings = evaluate(facts, baseline)
  writeJSON file: 'drift.json', pretty: 2,
            json: [host: facts.computerName, findings: findings]

  if (findings.isEmpty()) { return }

  findings.each { echo "${it.severity}: ${it.check} — ${it.detail}" }

  if (params.FAIL_ON_DRIFT) {
    unstable("${findings.size()} finding(s) against baseline")
  }
}

The distinction earns its keep. Unstable means the check ran and the machine is not what we said it should be. Failed means the check itself fell over and we currently know nothing. Those want different reactions from whoever is on the receiving end, and collapsing them into one red circle trains people to ignore both.

Same reasoning for the notifications: failure and fixed, never always. A job that emails on every run gets a mail rule within a week, and then the one that mattered goes into the same folder.

Fanning out

One agent became three, and a job per agent is three jobs to keep in step. nodesByLabel, from the Node and Label Parameter plugin, turns the label into the list:

stage('Every Windows agent') {
  steps {
    script {
      def names = nodesByLabel label: 'windows', offline: false
      parallel names.collectEntries { name ->
        [(name): {
          node(name) {
            checkout scm
            powershell script: 'ci\\collect.ps1 -OutFile facts.json',
                       label: "collect on ${name}"
            compare(readJSON(file: 'facts.json'),
                    readJSON(file: "ci/baseline/${name}.json"))
          }
        }]
      }
    }
  }
}

collectEntries over the list is safe because name is the closure's own parameter. The version that is not safe, and that everybody writes once, is the for loop:

// Every branch runs against the last agent in the list.
def branches = [:]
for (name in names) {
  branches[name] = { node(name) { ... } }
}

All three closures capture the same variable, and by the time they run it holds whatever the loop left in it. Copy it into a local inside the loop body, or use collectEntries and stop thinking about it.

The other thing to know is that this is Groovy running under a continuation-passing transform, and a chunk of what you think you know about closures and iteration does not hold. That is a whole post of its own, and it is the one I am writing next month.

What it actually caught

In the first three weeks, three things, none of which anybody had noticed.

A code-signing certificate with five weeks left on it, which is enough time to renew calmly and not enough to renew on the day of a release.

An agent whose C:\j Defender exclusion had gone. A group policy refresh had put the machine back to the domain baseline. Build times on that box had gone up by a third and everyone had decided the build was just slow now.

And one machine that had quietly dropped back to the default TLS settings after a patch cycle, which is the exact failure I wrote about a fortnight ago and had assumed was fixed permanently. It was fixed permanently on the machine I fixed it on.

What it cost

The pipeline is ninety lines of Groovy. The PowerShell behind it is two hundred and fourteen, in a .ps1 in the repository, and I cannot test any of it without a Windows box, a Jenkins agent and a night's patience.

That is the next problem, and it is why the interesting half of this project stops being Groovy in February.