Summary
scripts/vscode/Invoke-MSTestWithCoverage.ps1 exits non-zero for any -SearchRoot-scoped
invocation, even when every test in the scoped assembly passes, because its post-run gate
compares a repository-wide document-level Cobertura line-rate against a hard-coded 80%
threshold that a single-assembly run can never reach.
Environment
- OS/version: Windows 11 Pro 10.0.26200
- Command/flags used:
pwsh -NoProfile -Command './scripts/vscode/Invoke-MSTestWithCoverage.ps1 -SearchRoot QuickFiler.Test -Configuration Debug -CoverageOutput "coverage\coverage.cobertura.xml"; Write-Output "EXITCODE=$LASTEXITCODE"'
- Data source or fixture: repository
TaskMaster, branch bug/quickfiler-date-time-format-missing-invariant-culture-742 (also reproduces on the unfixed base tree, i.e. it is not caused by any in-flight change)
Steps to Reproduce
- From the repo root, run
./scripts/vscode/Invoke-MSTestWithCoverage.ps1 -SearchRoot QuickFiler.Test -Configuration Debug -CoverageOutput "coverage\coverage.cobertura.xml".
- Observe the vstest run itself completes with all
QuickFiler.Test tests passing (confirmed via the runner's own trx: 1429/1429 passed on the unfixed tree, 1434/1434 passed after an unrelated fix landed).
- Observe
$LASTEXITCODE is 1 regardless, and the console shows an exception thrown from Assert-CoberturaLineCoverageThreshold in scripts/vscode/Invoke-MSTestWithCoverage.Threshold.ps1 line 54: Cobertura line coverage 24.3028% is below the required 80% threshold.
Expected Behavior
A -SearchRoot-scoped coverage run's exit code should reflect the pass/fail state of the tests it actually executed (or the runner should measure coverage over the scoped assembly's own instrumented lines, or expose a way to skip/adjust the threshold for a deliberately narrow run). Callers that need "did my scoped test run pass" cannot get that signal from the exit code today.
Actual Behavior
Assert-CoberturaLineCoverageThreshold reads the line-rate attribute of the Cobertura document's root /coverage element, which spans every instrumented assembly in the solution regardless of -SearchRoot (61,894-61,898 valid lines observed), not just the assembly the scoped run executed. A -SearchRoot-scoped run only exercises and covers a fraction of that repository-wide denominator (15,040-15,047 lines observed, ~24.3%), so the hard-coded if ($percentage -lt 80) check at line 52 of Invoke-MSTestWithCoverage.Threshold.ps1 always throws for a scoped run, independent of test outcome. Invoke-MSTestWithCoverage.ps1 exposes only -SearchRoot, -Configuration, -CoverageOutput, and -NoExecute - no parameter lowers, disables, or rescales the threshold for scoped invocations.
Discovered while executing the approved atomic plan for issue #742: the plan's task [P5-T4] (and, before it, baseline task [P0-T8]) both intentionally scope -SearchRoot QuickFiler.Test to avoid a known unrelated hang in UtilitiesCS.Test's shell-icon test classes on this machine, and both runs hit this false-fail exit code even though the scoped test suite was 100% green in both cases. The task's literal acceptance criterion ("EXITCODE is 0") could not be satisfied, and the corresponding item in spec.md's Acceptance Criteria list (the criterion mapped to vstest.console.exe ... reports zero failures) was left unchecked as a result, even though that criterion is substantively true (1434/1434 passed).
Logs / Screenshots
Impact / Severity
Not a blocker because the unscoped, whole-repository invocation (without -SearchRoot) is unaffected - the document-level denominator then matches what is actually executed. It becomes a real obstacle any time a plan or workflow deliberately scopes a run (as this issue's plan did, to route around the separate UtilitiesCS.Test shell-icon hang), because the scoped run's exit code is then unusable as a pass/fail signal and every such plan must fall back to reading the trx directly.
Source
From: docs/features/potential/2026-09-14-coverage-runner-searchroot-threshold-false-fail.md
Summary
scripts/vscode/Invoke-MSTestWithCoverage.ps1exits non-zero for any-SearchRoot-scopedinvocation, even when every test in the scoped assembly passes, because its post-run gate
compares a repository-wide document-level Cobertura
line-rateagainst a hard-coded 80%threshold that a single-assembly run can never reach.
Environment
pwsh -NoProfile -Command './scripts/vscode/Invoke-MSTestWithCoverage.ps1 -SearchRoot QuickFiler.Test -Configuration Debug -CoverageOutput "coverage\coverage.cobertura.xml"; Write-Output "EXITCODE=$LASTEXITCODE"'TaskMaster, branchbug/quickfiler-date-time-format-missing-invariant-culture-742(also reproduces on the unfixed base tree, i.e. it is not caused by any in-flight change)Steps to Reproduce
./scripts/vscode/Invoke-MSTestWithCoverage.ps1 -SearchRoot QuickFiler.Test -Configuration Debug -CoverageOutput "coverage\coverage.cobertura.xml".QuickFiler.Testtests passing (confirmed via the runner's own trx: 1429/1429 passed on the unfixed tree, 1434/1434 passed after an unrelated fix landed).$LASTEXITCODEis1regardless, and the console shows an exception thrown fromAssert-CoberturaLineCoverageThresholdinscripts/vscode/Invoke-MSTestWithCoverage.Threshold.ps1line 54:Cobertura line coverage 24.3028% is below the required 80% threshold.Expected Behavior
A
-SearchRoot-scoped coverage run's exit code should reflect the pass/fail state of the tests it actually executed (or the runner should measure coverage over the scoped assembly's own instrumented lines, or expose a way to skip/adjust the threshold for a deliberately narrow run). Callers that need "did my scoped test run pass" cannot get that signal from the exit code today.Actual Behavior
Assert-CoberturaLineCoverageThresholdreads theline-rateattribute of the Cobertura document's root/coverageelement, which spans every instrumented assembly in the solution regardless of-SearchRoot(61,894-61,898 valid lines observed), not just the assembly the scoped run executed. A-SearchRoot-scoped run only exercises and covers a fraction of that repository-wide denominator (15,040-15,047 lines observed, ~24.3%), so the hard-codedif ($percentage -lt 80)check at line 52 ofInvoke-MSTestWithCoverage.Threshold.ps1always throws for a scoped run, independent of test outcome.Invoke-MSTestWithCoverage.ps1exposes only-SearchRoot,-Configuration,-CoverageOutput, and-NoExecute- no parameter lowers, disables, or rescales the threshold for scoped invocations.Discovered while executing the approved atomic plan for issue #742: the plan's task
[P5-T4](and, before it, baseline task[P0-T8]) both intentionally scope-SearchRoot QuickFiler.Testto avoid a known unrelated hang inUtilitiesCS.Test's shell-icon test classes on this machine, and both runs hit this false-fail exit code even though the scoped test suite was 100% green in both cases. The task's literal acceptance criterion ("EXITCODE is 0") could not be satisfied, and the corresponding item inspec.md's Acceptance Criteria list (the criterion mapped tovstest.console.exe ... reports zero failures) was left unchecked as a result, even though that criterion is substantively true (1434/1434 passed).Logs / Screenshots
docs/features/active/2026-09-02-quickfiler-date-time-format-missing-invariant-culture-742/evidence/baseline/vstest-coverage-baseline.2026-09-12T16-09.mdand.../evidence/qa-gates/vstest-coverage-final.2026-09-12T16-09.md.Impact / Severity
Not a blocker because the unscoped, whole-repository invocation (without
-SearchRoot) is unaffected - the document-level denominator then matches what is actually executed. It becomes a real obstacle any time a plan or workflow deliberately scopes a run (as this issue's plan did, to route around the separateUtilitiesCS.Testshell-icon hang), because the scoped run's exit code is then unusable as a pass/fail signal and every such plan must fall back to reading the trx directly.Source
From: docs/features/potential/2026-09-14-coverage-runner-searchroot-threshold-false-fail.md