Skip to content

docs: Fix the AI migration prompt against real consumer builds - #227

Open
tablackburn wants to merge 1 commit into
mainfrom
docs/159-test-migration-prompt
Open

docs: Fix the AI migration prompt against real consumer builds#227
tablackburn wants to merge 1 commit into
mainfrom
docs/159-test-migration-prompt

Conversation

@tablackburn

Copy link
Copy Markdown
Contributor

Summary

The migration guide's AI prompt is a v1.0.0 deliverable that had never been run. I ran it verbatim against four real public consumer builds and fixed the five defects it surfaced. This is the "test the AI prompt" half of #159 only — the guide coherence pass and the changelog entry are untouched.

Consumers tested, unmodified, each migrated by a fresh agent given only the prompt and the repository:

  • jimbrig/PSUtils — psake; has the bogus Test.ScriptAnalysisEnabled, floating latest pins
  • flycastpartnersinc/FreshservicePS — psake; pins PowerShellBuild 0.6.1, psake 4.9.0, Pester 5.3.3
  • pauby/PSTodoWarrior — Invoke-Build; non-default .pstodowarrior.build.ps1, inline PSDepend hashtable
  • milestonesys/MilestonePSTools — psake; many $PSBPreference values, AlphabeticParamsOrder, bootstrap manifest

What the prompt got right

The fixes are narrow, so this is worth recording:

  • No run missed a documented break, invented a change the guide does not call for, or produced a file that fails to parse (re-verified with the PowerShell AST parser).
  • The anti-summarization instruction works. Every run explicitly declined WebFetch, citing its summarizing behaviour, and fetched raw text instead.
  • Step 7 produced the highest-value findings — breaks no entry covers. Best example: MilestonePSTools' SetOnlineHelpUrls task matches ^online version: in docs front matter, which PlatyPS 1.x renames to HelpUri:; unflagged, every command doc silently loses its online-help URL.
  • Step 8's checklist framing works — every run returned the full entry checklist, not just its diff.

Defects fixed

  1. Step 5's version-pin syntax was wrong for most consumers. It named -Version on task ... -FromModule PowerShellBuild. Three of four fixtures use -MinimumVersion, and -Version is an alias for -RequiredVersion — following the step literally turns a floor into an exact-version requirement. Every psake run flagged it and improvised. Now: keep whichever parameter the consumer already uses, change only its value.
  2. Step 5 broke floating pins. Two runs replaced PowerShellBuild = 'latest' with a hard '1.0.0', breaking the bootstrap immediately where latest would have kept working and picked up 1.0.0 on release. Now: leave floating pins floating and report them.
  3. The baseline step contradicted its own position — "Before editing anything" was numbered after the step that applies edits. Now step 2.
  4. $PSBPreference was said to be "populated in build.ps1". It is set in the psake file's properties block or in .build.ps1; build.ps1 is a bootstrap wrapper. The inputs section now points at the task file and allows for names like .mymodule.build.ps1.
  5. The completeness self-check only checked the tail. One run computed "16 entries matching 16 bullets" — wrong on both sides, self-consistent, and it passed its own check. The check now pairs each Quick Start bullet with its heading rather than counting, and step 8 reuses that list as the reporting checklist.

Plus a one-line non-interactive fallback for the prompt's three "ask me" branches, and a correction to the claim about what the prompt was tested against.

Test plan

  • Prompt run verbatim against four real consumer builds; every edit reviewed against the guide
  • Migrated files parse cleanly ([Parser]::ParseFile, Import-PowerShellDataFile)
  • Revised prompt re-run against the same consumer: -MinimumVersion preserved, floating pin left floating with a marker, all 18 entries paired and reported
  • No trailing whitespace; added lines within the line-length limit

Breaking changes

None. Documentation only.

Refs #159

The migration guide's AI prompt had never been exercised. Running it
verbatim against four real public consumer builds (three psake, one
Invoke-Build, PowerShellBuild pins from 0.6.1 to 0.8.2) surfaced five
defects. None produced a build file that would not parse, but two could
silently change build behaviour.

- Step 5 told the agent to change "the -Version on `task ... -FromModule
  PowerShellBuild -Version`". Most real consumers use -MinimumVersion;
  -Version is an alias for -RequiredVersion, so following the step
  literally converts a floor into an exact-version requirement. Every
  psake run flagged this as wrong and had to improvise. The step now says
  to keep whichever parameter the consumer already uses and change only
  its value.
- Step 5 also drove agents to replace floating pins ('latest') with a
  hard pin on the unpublished 1.0.0, breaking the consumer's bootstrap
  immediately where 'latest' would have kept working. It now says to
  leave floating pins floating and report them.
- The baseline step opened with "Before editing anything" but was
  numbered after the step that applies edits. It is now step 2.
- The conventions section said $PSBPreference is "populated in
  build.ps1". It is set in the psake file's properties block or in
  .build.ps1; build.ps1 is only a bootstrap wrapper. The inputs section
  now points at the task file and allows for names like
  .mymodule.build.ps1.
- The completeness self-check only verified the document's tail. One run
  computed a self-consistent but wrong entry count and still passed it.
  The check now pairs every Quick Start bullet with its heading, and
  step 8 uses that list as the reporting checklist.

Also adds a non-interactive fallback for the prompt's three "ask me"
branches, and corrects the claim about what the prompt was tested
against.

Re-running the revised prompt against the same consumer confirms each
fix: the -MinimumVersion parameter is preserved, the floating pin is left
floating with a review marker, and all 18 entries are paired and
reported.

Refs #159

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011GYJrhbrDzqufaeMqD9QjT
Copilot AI lite review requested due to automatic review settings August 28, 2026 22:14

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@github-actions

Copy link
Copy Markdown

Test Results

    4 files  ±0    889 suites  ±0   2m 37s ⏱️ +10s
  596 tests ±0    593 ✅ ±0   3 💤 ±0  0 ❌ ±0 
2 367 runs  ±0  2 295 ✅ ±0  72 💤 ±0  0 ❌ ±0 

Results for commit 4c35102. ± Comparison against base commit 3ea9e82.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants