The psake/Invoke-Build task drift guard compares nothing against nothing

[tablackburn]

tests/IBTasks.tests.ps1, the It 'Contains all the tasks that were in the Psake file', is the only guard that psake and Invoke-Build still define the same tasks. It cannot fail. Two independent defects cancel each other out, and fixing either one alone turns it red.

What happens

Defect one: $IBTasksResult is $null inside the It that reads it.

$IBTasksResult = $null                       # Describe body: runs at discovery only
It 'Parseable by invoke-build' {
    $IBTasksResult = Start-Job { ... } | Wait-Job | Receive-Job    # assigned here
    ...
}
It 'Contains all the tasks that were in the Psake file' {
    $IBTaskNames = $IBTasksResult.all.name   # ...and $null here

Each It block runs in its own scope, so the assignment in the first block is not visible in the second, and the $IBTasksResult = $null at the Describe level runs during discovery rather than during the run. $IBTaskNames is therefore $null.

Defect two: $psakeTaskNames is a collection of $null.

$psakeTaskNames = Start-Job -ScriptBlock {
    Invoke-PSake -docs -buildfile $using:psakeFilePath | Where-Object name -notmatch '^(default|\?)$' | ForEach-Object name
} | Wait-Job | Receive-Job

Invoke-PSake -docs formats a table to the output stream. What crosses the job boundary is therefore format records, not task objects, and a format record has no Name property. Measured on this repository at 484f24f:

Deserialized.Microsoft.PowerShell.Commands.Internal.Format.FormatStartData   1
Deserialized.Microsoft.PowerShell.Commands.Internal.Format.GroupStartData    1
Deserialized.Microsoft.PowerShell.Commands.Internal.Format.FormatEntryData  17
Deserialized.Microsoft.PowerShell.Commands.Internal.Format.GroupEndData      1
Deserialized.Microsoft.PowerShell.Commands.Internal.Format.FormatEndData     1

$psakeTaskNames.Count  = 21
non-null entries       = 0

Where-Object name -notmatch ... also passes all 21 through, because $null -notmatch anything is true.

The two cancel. The loop becomes foreach ($taskItem in @($null, $null, ...)), the test is $null -notin $null, which evaluates to False, so the throw never fires. The closing assertion is @($null, $null, ...) | Should -Not -BeNullOrEmpty, which passes because the array has 21 elements even though every one of them is $null.

Measured: the guard does not fire on the divergence it exists to catch

Renaming a single task in the built IB.tasks.ps1 so it no longer matches psakeFile.ps1 — exactly the drift the test was written for:

-Task Sign SignModule, SignCatalog
+Task RenamedSignTask SignModule, SignCatalog

Describing Invoke-Build Tasks
  [+] IB.tasks.ps1 exists
  [+] Parseable by invoke-build
  [+] Contains all the tasks that were in the Psake file      <-- should be red
Tests Passed: 8, Failed: 0

PowerShell 7.6.5, Pester 6.1.0, psake 5.0.4, Windows 11.

Why it matters

Divergence between the two task files is not hypothetical; it is this repository's most productive defect class, and it reaches Invoke-Build consumers only.

• IB.tasks.ps1 reads a nonexistent CodeCoverage.OutputFormat setting, so Invoke-Build users get $null #178: IB.tasks.ps1 read Test.CodeCoverage.OutputFormat while the defaults define OutputFileFormat, so no Invoke-Build consumer had a working code coverage format.
• $PSBPreference.Sign.SkipCertificateValidation has no effect on the Store and Thumbprint sources it is documented for #193: IB.tasks.ps1 never passed SkipValidation, so $PSBPreference.Sign.SkipCertificateValidation was dead for every Invoke-Build consumer while it worked for psake consumers.

Both were found by hand and then covered by the other Describe blocks in this same file — the settings-path comparison and the signing-settings comparison — which do work. The task-name comparison, the oldest of the three and the one everything else was layered on top of, has never worked. A task added to psakeFile.ps1 and forgotten in IB.tasks.ps1 would ship silently today.

Options

1. Fix both defects and keep the test. Move the whole thing into one It (or hoist the Invoke-Build run into a BeforeAll that assigns to a $script: variable), and get real task names out of psake with Get-PSakeScriptTasks -buildFile $psakeFilePath instead of Invoke-PSake -docs, which returns objects rather than a formatted table. Both halves must change together; either alone makes the test fail for the wrong reason.
2. Replace it with a static comparison, like its two neighbours. The Settings referenced by the task files and Signing settings referenced by the task files blocks in this same file parse the two task files with a regular expression and compare, with no job, no psake invocation, and no format-record hazard. Task names could be compared the same way, which would also make the test fast and immune to psake's output format changing again.
3. Delete it. It has never tested anything, and the two working Describe blocks below it cover the settings-level drift that has actually bitten. This is the cheapest option and the one that loses real coverage, since neither of those blocks can see a task that exists in one file and not the other.

(2) looks like the best value: it keeps the coverage, matches the pattern the rest of the file already uses, and removes the job entirely.

Related: #178, #193.