Skip to content

aitools: report a Windows shim that printed help but exited non-zero under its own reason - #7476

Merged
anton-107 merged 1 commit into
mainfrom
aitools/windows-shim-probe
Nov 12, 2026
Merged

anton-107 merged 1 commit into
mainfrom
aitools/windows-shim-probe

Conversation

@anton-107

@anton-107 anton-107 commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Stacked on #7475.

Stacked on: aitools: typed fetch, state and agent CLI timeout errors

Summary

On Windows, the codex.cmd and copilot.bat shims can print the full plugin --help and then exit 1. Because of that, databricks aitools install reported a CLI that is present as cli-not-on-path, which points the user at the wrong fix. The probe now reports that case as shim-exited-with-error and still stops before running any plugin command:

 probePluginCLI(agent)
   bin = lookPath(agent CLI)
   err = run `<bin> plugin --help` (20s timeout)
-  if err: block cli-not-on-path
+  if err and windowsShimPrintedHelp(bin, err): block shim-exited-with-error
+      ("printed its plugin help but exited with an error (<exit>)[: <stderr>];
+        to see the error and exit code, run
+        `& "<bin>" plugin --help; $LASTEXITCODE` in PowerShell or
+        `"<bin>" plugin --help` and then `echo %ERRORLEVEL%` in Command Prompt, and
+        re-run this command once it exits with 0")
+  if err: block cli-not-on-path

+windowsShimPrintedHelp(bin, err)
+  bin ends in .cmd or .bat (LookPath returns these only on Windows)
+  and err is not a timeout
+  and stdout has a `Usage: <bin> plugin` line
+      and indented `install`/`add` and `marketplace` command lines

blockedErrorCategory maps the new reason to PLUGIN_INSTALL_FAILED, the closest existing category.

Changelog fragment: .nextchanges/cli/aitools-windows-shim-probe.md.

Evidence

Tests in libs/aitools/installer/plugin_test.go. The stubbed lookPath returns .cmd/.bat paths, so the Windows path runs on any platform. plugin_windows_test.go also runs a real .cmd that prints the help and exits 1, on the Windows CI cells only.

  • Before (shim check removed from probePluginCLI, otherwise this head):
    --- FAIL: TestProbeWindowsShimPrintedHelp/windows_.bat_shim_printed_help_and_exited_1
    --- FAIL: TestProbeWindowsShimPrintedHelp/windows_.cmd_shim_printed_help_and_exited_1
    --- FAIL: TestProbeWindowsShimPrintedHelp/windows_shim_under_a_path_with_a_space
    --- FAIL: TestProbeWindowsShimPrintedHelp/windows_shim_that_printed_an_error_on_stderr
    --- FAIL: TestProbeWindowsShimPrintedHelp/windows_codex_.cmd_shim_printed_clap_help_and_exited_1
        expected: "shim-exited-with-error"   actual: "cli-not-on-path"
    
    After (this head):
    --- PASS: TestProbeWindowsShimPrintedHelp (12 cases: Copilot .bat/.cmd, a path with a
              space, stderr kept in the detail, and Codex .cmd blocked as
              shim-exited-with-error with no plugin command run; help on stderr,
              no plugin command, unix binary, .exe, error text mentioning the words,
              usage line without command list, unindented words all still cli-not-on-path)
    --- PASS: TestWindowsShimPrintedHelpRejectsTimeout
    --- PASS: TestClassifyInstallError/blocked_shim_exited_with_error (PLUGIN_INSTALL_FAILED)
    --- PASS: TestProbeRealCmdShimPrintedHelp (Windows CI cells: a real .cmd exits 1 → shim-exited-with-error)
    

Merge Danger

Door: two-way

Only a code path changes. No state or config changes. In --output json, this case now reports error_category PLUGIN_INSTALL_FAILED instead of CLI_NOT_ON_PATH.

Blast Radius: narrow

  • Behavior change on Windows only: a .cmd/.bat agent CLI whose plugin --help exits non-zero but prints the plugin help is now blocked as shim-exited-with-error instead of cli-not-on-path, with the command to run to see the error. As before, no marketplace or plugin command runs. On every other platform, for .exe, and for timeouts, the probe behaves as before.
  • Exit codes and network waits do not change.
  • Not tested on a real Windows machine or on macOS. The help detection is matched against the help format of the current agent CLIs, so if a CLI changes that format, the probe falls back to cli-not-on-path.

This pull request and its description were written by Isaac.

@anton-107
anton-107 requested review from a team and lennartkats-db as code owners November 11, 2026 14:26
@anton-107 anton-107 added the ai-auto PRs made by AI loops & factories, label Nov 11, 2026
@anton-107
anton-107 requested a review from a team as a code owner November 11, 2026 14:26
@anton-107
anton-107 requested a review from misha-db November 11, 2026 14:26
@anton-107
anton-107 force-pushed the aitools/typed-install-errors branch from 9132948 to 405e077 Compare November 11, 2026 14:33
@anton-107
anton-107 requested a review from a team as a code owner November 11, 2026 14:33
@anton-107
anton-107 force-pushed the aitools/windows-shim-probe branch from 5601cd2 to 2c28258 Compare November 11, 2026 14:33
@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Collaborator

✅ Integration test report

Commit: b28c0c8

Run: 43219555848

Env 🔄​flaky 💚​RECOVERED ✅​pass 🙈​skip Time
🔄​ aws linux-2core-8gb 2 1 268 27 14:05
💚​ aws-windows-latest-4core-16gb 1 272 25 11:48
💚​ azure linux-2core-8gb 1 269 27 16:38
🔄​ azure-windows-latest-4core-16gb 1 1 270 25 11:35
💚​ gcp linux-2core-8gb 1 270 27 16:57
🔄​ gcp-windows-latest-4core-16gb 1 1 271 25 17:00
Test Name aws linux-2core-8gb aws-windows-latest-4core-16gb azure linux-2core-8gb azure-windows-latest-4core-16gb gcp linux-2core-8gb gcp-windows-latest-4core-16gb
💚​ TestAccept 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
🔄​ TestSyncFullFileSync ✅​p ✅​p ✅​p ✅​p ✅​p 🔄​f
🔄​ TestSyncIncrementalSyncFileToPythonNotebook 🔄​f ✅​p ✅​p ✅​p ✅​p ✅​p
🔄​ TestSyncIncrementalSyncPythonNotebookToFile ✅​p ✅​p ✅​p 🔄​f ✅​p ✅​p
🔄​ TestSyncNestedFolderSync 🔄​f ✅​p ✅​p ✅​p ✅​p ✅​p
Top 34 slowest tests (at least 2 minutes):
duration env testname
5:06 gcp-windows-latest-4core-16gb TestFilerWorkspaceFilesExtensionsDelete
3:58 azure linux-2core-8gb TestAccept
3:44 aws linux-2core-8gb TestAccept
3:43 aws-windows-latest-4core-16gb TestAccept
3:42 azure-windows-latest-4core-16gb TestAccept
3:41 gcp-windows-latest-4core-16gb TestAccept
3:21 gcp linux-2core-8gb TestAccept
3:14 aws-windows-latest-4core-16gb TestFilerWorkspaceFilesExtensionsRead
3:09 gcp linux-2core-8gb TestFilerWorkspaceFilesExtensionsReadDir
3:05 aws-windows-latest-4core-16gb TestFilerRecursiveDelete/workspace_files
2:58 gcp-windows-latest-4core-16gb TestFsCpFileToFileWithOverwriteFlag/dbfs_to_dbfs
2:51 aws-windows-latest-4core-16gb TestFilerReadWrite/workspace_files_extensions
2:46 gcp linux-2core-8gb TestFilerRecursiveDelete/dbfs
2:42 azure linux-2core-8gb TestFilerReadDir/workspace_files
2:41 gcp-windows-latest-4core-16gb TestFsCpDirToDirWithOverwriteFlag/local_to_dbfs
2:35 gcp linux-2core-8gb TestFilerWorkspaceFilesExtensionsRead
2:33 gcp-windows-latest-4core-16gb TestFsCpFileToFileWithOverwriteFlag/dbfs_to_uc-volumes
2:22 gcp-windows-latest-4core-16gb TestFsCpDirToDirWithOverwriteFlag/dbfs_to_dbfs
2:22 gcp-windows-latest-4core-16gb TestFsCpDirToDirWithOverwriteFlag/dbfs_to_local
2:20 gcp-windows-latest-4core-16gb TestFilerWorkspaceFilesExtensionsReadDir
2:15 gcp linux-2core-8gb TestFilerWorkspaceFilesExtensionsStat
2:13 gcp linux-2core-8gb TestFsCpDirToDirFileNotOverwritten/dbfs_to_local
2:12 azure-windows-latest-4core-16gb TestFsCpDirToDirWithOverwriteFlag/dbfs_to_dbfs
2:11 gcp-windows-latest-4core-16gb TestImportDirDoesNotOverwrite
2:08 azure linux-2core-8gb TestFilerWorkspaceFilesExtensionsReadDir
2:07 aws-windows-latest-4core-16gb TestImportDirWithOverwriteFlag
2:07 azure-windows-latest-4core-16gb TestFilerWorkspaceFilesExtensionsReadDir
2:05 azure linux-2core-8gb TestFsCpDirToDirFileNotOverwritten/dbfs_to_dbfs
2:05 aws linux-2core-8gb TestFilerWorkspaceFilesExtensionsRead
2:04 gcp linux-2core-8gb TestFsCpDir/dbfs_to_uc-volumes
2:03 azure-windows-latest-4core-16gb TestWorkspaceFilesExtensionsNotebooksAreNotStatAsFiles
2:02 gcp-windows-latest-4core-16gb TestFilerReadWrite/dbfs
2:02 gcp-windows-latest-4core-16gb TestFilerReadDir/workspace_files
2:00 aws-windows-latest-4core-16gb TestFsCpFileToFileWithOverwriteFlag/dbfs_to_dbfs

@anton-107
anton-107 force-pushed the aitools/typed-install-errors branch from 405e077 to 1aa2b9f Compare November 11, 2026 16:53
@anton-107
anton-107 force-pushed the aitools/windows-shim-probe branch from 2c28258 to 4a70459 Compare November 11, 2026 16:53
@anton-107
anton-107 force-pushed the aitools/typed-install-errors branch from 1aa2b9f to bbd0a95 Compare November 11, 2026 16:54
@anton-107
anton-107 force-pushed the aitools/windows-shim-probe branch from 4a70459 to c162e8e Compare November 11, 2026 16:54
@anton-107
anton-107 force-pushed the aitools/typed-install-errors branch from bbd0a95 to f18c292 Compare November 11, 2026 16:57
@anton-107
anton-107 force-pushed the aitools/windows-shim-probe branch from c162e8e to 0c0d208 Compare November 11, 2026 16:57

@simonfaltum simonfaltum left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review from simonfaltum's review agent (pr-review skill) at 0c0d208. Each finding was verified against the code; simonfaltum may follow up.

Not approving yet: 1 blocking issue(s), tagged [blocking]; 1 open design question.

This PR makes the plugin probe accept a Windows .cmd/.bat agent CLI that prints its plugin --help and then exits non-zero, instead of reporting it as cli-not-on-path. The gate is narrow and the new tests pass. Two things should be resolved before merge. First, the three help regexes are compiled in package-level vars, which the repo's Go style guide forbids. Second, the fix rests on a cause nobody has verified: if the shim exits non-zero on every command, install now fails at a later step and leaves a marketplace and plugin that aitools doesn't track. The rest are test-coverage, comment and changelog fixes.

Questions

  1. Do codex.cmd and copilot.bat exit non-zero only on plugin --help, or on every command that runs through the shim? The change hasn't run on Windows, and the tests use a hand-written help string. If every call exits non-zero, accepting the probe means marketplace add and plugin install now run, report install-failed, and leave a marketplace and plugin that aitools doesn't track. Today the probe stops before anything changes. Could you capture the real stdout, stderr and exit codes on Windows for plugin --help and for a plugin install through each shim, and use that output as the test fixture? If the shim layer turns out to be the cause, would it fit better to handle it in runAgentCmd, so install, update and uninstall all benefit? Another option is to report this case under its own reason instead of cli-not-on-path.

Comments outside the changed lines

  • [nit] .github/PULL_REQUEST_TEMPLATE.md: The description uses ## Summary, ## Evidence and ## Merge Danger. .agents/skills/pr-checklist/SKILL.md asks agent-written PRs to use the template's ## Changes, ## Why and ## Tests, in that order. It names ## Summary as a section not to add, and asks for the agent disclosure as a single italic last line. Move the content under the template headings: Evidence fits under Tests, and Merge Danger under Why.

Comment thread libs/aitools/installer/plugin.go Outdated
Comment on lines +283 to +285
pluginHelpUsage = regexp.MustCompile(`(?im)^usage:[ \t]+\S+[ \t]+plugin\b`)
pluginHelpInstall = regexp.MustCompile(`(?m)^[ \t]+(install|add)\b`)
pluginHelpMarket = regexp.MustCompile(`(?m)^[ \t]+marketplace\b`)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[blocking] These three regexp.MustCompile calls run in package-level var initializers, which .agents/rules/style-guide-go.md forbids ("Do not compile regexps ... in package-level var initializers or init()"). The main binary links libs/aitools/installer, so every databricks command on every OS compiles three patterns. Only the Windows .cmd/.bat path uses them, after a failed probe. Wrap each one in sync.OnceValue(func() *regexp.Regexp { return regexp.MustCompile(...) }) as cmd/air/runconfig.go:31 does. Then call them only after the GOOS, extension and timeout checks in windowsShimPrintedHelp. The new tests already go through that function, so a bad pattern still panics in tests.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in e158591: the three patterns are now sync.OnceValue funcs (same shape as cmd/air/runconfig.go), and windowsShimPrintedHelp calls them only after the GOOS, extension, timeout and ProcessError checks. The table test exercises all three, so a bad pattern still panics in tests.

Comment thread libs/aitools/installer/plugin_test.go Outdated
accept bool
}{
{name: "windows .bat shim printed help and exited 1", goos: "windows", bin: `C:\npm\copilot.bat`, stdout: copilotPluginHelp, accept: true},
{name: "windows .cmd shim printed help and exited 1", goos: "windows", bin: `C:\npm\codex.cmd`, stdout: copilotPluginHelp, accept: true},

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nice to have] The .cmd case uses a codex.cmd path but gives the Copilot agent Copilot's commander-style help, which lists install. Codex prints clap-style help (Usage: codex plugin [OPTIONS] <COMMAND>) with an add subcommand. So the add branch of pluginHelpInstall, which exists only for Codex, is never tested. If that branch or the Codex help shape broke, every test would still pass, and codex.cmd would go back to reporting cli-not-on-path. Add a row with codexAgent() and Codex's real plugin --help output, ideally captured on Windows.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added in e158591: a codexAgent() row with codex.cmd and clap-style help (Usage: codex plugin [OPTIONS] <COMMAND> listing add and marketplace), so the add branch is covered. The former .cmd row now uses copilot.cmd, so it no longer mixes Codex's path with Copilot's help. The Codex fixture is a representative shape, not output captured on Windows; that capture belongs to the open Question 1.

Comment thread libs/aitools/installer/plugin.go Outdated
// probeGOOS is runtime.GOOS; tests replace it to exercise the Windows path.
var probeGOOS = runtime.GOOS

// pluginHelpUsage and pluginHelpCommand match the help that `<agent> plugin

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nit] The comment says "pluginHelpUsage and pluginHelpCommand match…", but there is no pluginHelpCommand; the variables are pluginHelpUsage, pluginHelpInstall and pluginHelpMarket. In the windowsShimPrintedHelp doc below, line 290 runs to about 92 columns while the lines around it wrap near 80. That doc's second half also lists the guard clauses the code already shows. Fix the name, rewrap the line, and keep only the reason for the quirk.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in e158591: the comment names pluginHelpUsage, pluginHelpInstall and pluginHelpMarket (noting add for Codex), and the windowsShimPrintedHelp doc is rewrapped and keeps only the reason for the quirk.

Comment thread libs/aitools/installer/plugin_test.go Outdated
assert.False(t, windowsShimPrintedHelp(bin, &agentCmdTimeoutError{timeout: time.Second, err: perr}))
}

func TestProbeRejectsCLIWithoutPluginCommand(t *testing.T) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nit] This test leaves probeGOOS at the host value and uses stubAgentLookPath, which returns /usr/bin/codex with no extension. So windowsShimPrintedHelp returns false at the GOOS or extension check, before any regex runs, and the test passes on the base commit too. Make it a row in the TestProbeWindowsShimPrintedHelp table with goos: "windows", a .cmd bin and the unknown command 'plugin' output. Then it actually checks that the matcher rejects an error.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in e158591: TestProbeRejectsCLIWithoutPluginCommand is gone; it is now the windows shim without a plugin command row in TestProbeWindowsShimPrintedHelp (goos: "windows", C:\npm\codex.cmd, unknown command 'plugin' output), so it gets past the GOOS and extension checks and tests the matcher.

Comment thread libs/aitools/installer/plugin.go Outdated
return "", &BlockedError{Agent: agent.Name, Reason: ReasonCLINotOnPath, Detail: err.Error()}
}
if _, err := runAgentCmd(ctx, pluginProbeTimeout, []string{bin, "plugin", "--help"}); err != nil {
if _, err := runAgentCmd(ctx, pluginProbeTimeout, []string{bin, "plugin", "--help"}); err != nil && !windowsShimPrintedHelp(bin, err) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nit] When windowsShimPrintedHelp accepts a failed probe, the error is dropped without a log. If a later step fails, debug logs show nothing about the probe's non-zero exit, so Windows reports are hard to diagnose. Log it at debug level in probePluginCLI, where ctx is available, e.g. log.Debugf(ctx, "accepting %s plugin --help that exited non-zero after printing help: %v", bin, err).

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in e158591: probePluginCLI now logs accepting %s plugin --help that exited non-zero after printing help: %v at debug level when it accepts the failed probe.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Update in f6b4ec8: following the decision on Question 1, a shim that printed the help and exited non-zero is now blocked as shim-exited-with-error instead of accepted. So the debug log is gone; the exit status is in the error's detail that the user sees.

@@ -0,0 +1 @@
* `databricks aitools install` on Windows accepts a coding agent CLI shim (`.cmd` or `.bat`) that prints its plugin help and then exits with an error, instead of reporting the CLI as not on PATH. ([#7476](https://github.com/databricks/cli/pull/7476))

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nit] The entry names only databricks aitools install, but aitools update reaches the same probe through its legacy raw-skills migration (cmd/aitools/update.go:253 calls InstallPluginForAgent). The earlier probe-timeout fragment listed both commands. pr-checklist also asks a command-led entry to say "now". Suggest: "* databricks aitools install and databricks aitools update now accept a coding agent CLI shim (.cmd or .bat) on Windows that prints its plugin help and then exits with an error, instead of reporting the CLI as not on PATH. (#7476)"

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in e158591: the fragment now uses your wording, naming both databricks aitools install and databricks aitools update and keeping the PR link.

@anton-107
anton-107 force-pushed the aitools/typed-install-errors branch from f18c292 to af32385 Compare November 11, 2026 17:31
@anton-107
anton-107 force-pushed the aitools/windows-shim-probe branch from 0c0d208 to 873e0e0 Compare November 11, 2026 17:31
@anton-107
anton-107 force-pushed the aitools/typed-install-errors branch from af32385 to 2408150 Compare November 11, 2026 19:28
@anton-107
anton-107 force-pushed the aitools/windows-shim-probe branch 2 times, most recently from eabdd30 to e158591 Compare November 11, 2026 19:30
@anton-107

Copy link
Copy Markdown
Contributor Author

Thanks. Pushed e158591: the [blocking] regexps are now sync.OnceValue funcs, and all five other inline comments are addressed (replies in each thread).

On the PR description: I kept the current sections. This stack uses the same Summary / Evidence / Merge Danger layout on every PR, so reviewers can read them all the same way. The content covers what the template asks for (what changed, why, and how it was tested).

Question 1 (real Windows shim behavior and where to handle it) is still open. I'll follow up on it separately.

@anton-107
anton-107 force-pushed the aitools/windows-shim-probe branch from e158591 to f6b4ec8 Compare November 11, 2026 20:07
@anton-107 anton-107 changed the title aitools: accept a Windows shim probe that printed help but exited non-zero aitools: report a Windows shim that printed help but exited non-zero under its own reason Nov 11, 2026
@anton-107

Copy link
Copy Markdown
Contributor Author

On Question 1: we couldn't verify how codex.cmd and copilot.bat exit on Windows, so the safer change is to surface this case distinctly rather than run installs that may half-complete. Pushed f6b4ec8:

  • A .cmd/.bat shim that printed its plugin help but exited non-zero is now blocked under its own reason, shim-exited-with-error, instead of cli-not-on-path. The probe still stops before any marketplace add or plugin install, so nothing is left behind that aitools doesn't track.
  • The detail says what happened and what to do: <agent> printed its plugin help but exited with an error (exit status 1); run `<bin> plugin --help` to see the error, and re-run this command once it exits cleanly.
  • Title, commit subject, changelog fragment and description are updated to match. The test table now asserts the new reason and that only the probe ran.

Since the install no longer goes ahead, this doesn't touch runAgentCmd. If real Windows output later shows the shim exits non-zero on every command, handling it there would be a separate change.

@anton-107
anton-107 force-pushed the aitools/typed-install-errors branch from 2408150 to 5275455 Compare November 11, 2026 21:04
@anton-107
anton-107 force-pushed the aitools/windows-shim-probe branch from f6b4ec8 to 3108ec5 Compare November 11, 2026 21:04
@anton-107
anton-107 force-pushed the aitools/typed-install-errors branch from 2d5e5ca to 64a25aa Compare November 12, 2026 10:01
@anton-107
anton-107 force-pushed the aitools/windows-shim-probe branch from 6731eb1 to a7cd2a4 Compare November 12, 2026 10:01
@anton-107
anton-107 force-pushed the aitools/typed-install-errors branch from 64a25aa to 7f0dcdf Compare November 12, 2026 10:25
@anton-107
anton-107 force-pushed the aitools/windows-shim-probe branch from a7cd2a4 to d0cb9df Compare November 12, 2026 10:26

@simonfaltum simonfaltum left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Approval from simonfaltum's review agent at d0cb9df, on Simon's decision.

The design questions from the earlier automated reviews no longer block this PR: your answers are accepted, and any still-open ones are non-blocking. The changes since the last automated review (149851e → d0cb9df) address its comments.

@rugpanov rugpanov left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also found by our reviewers, already asked in simonfaltum's review (Question 1): the new reason fires only for a .cmd or .bat CLI whose help matches the three regexes. Our first-round notes asked the same: a split with no help parsing (the CLI was found, but plugin --help failed) gives the same signal on every platform.

Comment thread libs/aitools/installer/plugin.go Outdated
if stderr := strings.TrimSpace(perr.Stderr); stderr != "" {
detail += ": " + stderr
}
return detail + fmt.Sprintf("; to see the error and exit code, run `& \"%[1]s\" plugin --help; $LASTEXITCODE` in PowerShell or `\"%[1]s\" plugin --help & echo %%ERRORLEVEL%%` in Command Prompt, and re-run this command once it exits with 0", bin)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nice to have] In the Command Prompt form, "<bin>" plugin --help & echo %ERRORLEVEL%, cmd.exe expands %ERRORLEVEL% when it reads the line, before the shim runs. So it prints the exit code of the previous command, usually 0. The hint then says to re-run "once it exits with 0", so the user can think that the shim is fixed when it is not. Tell the user to run echo %ERRORLEVEL% as a separate command, and update the expected text in TestProbeWindowsShimPrintedHelp.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 735616f. The Command Prompt form now has two separate commands: "<bin>" plugin --help, then echo %ERRORLEVEL%. The second one reads the shim's exit code after it ran, not the code from when the line was read. The shimExitedDetail doc says why, and the expected text in TestProbeWindowsShimPrintedHelp is updated. The PowerShell form (& "<bin>" plugin --help; $LASTEXITCODE) stays as it was, since PowerShell evaluates $LASTEXITCODE after the call.

Comment thread libs/aitools/installer/plugin.go Outdated
// the agent CLIs print --help to stdout, and help on stderr usually means the
// CLI rejected its arguments.
//
// No public upstream report documents this exit code, so there is no link to

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nit] "No public upstream report documents this exit code, so there is no link to cite" answers a review question. It does not tell the caller anything. .agents/rules/comments.md asks for the contract only. Keep the pointer to TestProbeRealCmdShimPrintedHelp, and remove the rest of this paragraph.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 735616f (now c3363cb after a restack): the "no public upstream report" sentence is gone. The paragraph now only says that TestProbeRealCmdShimPrintedHelp runs a real .cmd that prints the help and exits 1, which checks how this path sees a shim's exit code on Windows.

@anton-107
anton-107 force-pushed the aitools/typed-install-errors branch from 7f0dcdf to 45cb4be Compare November 12, 2026 12:27
@anton-107
anton-107 force-pushed the aitools/windows-shim-probe branch 2 times, most recently from 1988e05 to 735616f Compare November 12, 2026 12:31
@anton-107
anton-107 force-pushed the aitools/typed-install-errors branch from 45cb4be to dccb816 Compare November 12, 2026 12:31
@anton-107
anton-107 force-pushed the aitools/windows-shim-probe branch from 735616f to e26c489 Compare November 12, 2026 12:31
@anton-107

Copy link
Copy Markdown
Contributor Author

Thanks. On the generic probe-failure reason: that's Question 1 from simonfaltum's review, answered in #7476 (comment). This PR stays limited to Windows .cmd/.bat shims that print their plugin help. A reason that fires whenever plugin --help fails after LookPath finds the CLI changes reporting on every platform, so it fits better as a follow-up.

Also in 735616f: the %ERRORLEVEL% fix (reply inline), and the windowsShimPrintedHelp doc comment now describes what the Windows test checks instead of answering a review question.

@anton-107
anton-107 force-pushed the aitools/typed-install-errors branch from dccb816 to 4deb804 Compare November 12, 2026 12:36
@anton-107
anton-107 force-pushed the aitools/windows-shim-probe branch from e26c489 to b28c0c8 Compare November 12, 2026 12:36
@anton-107
anton-107 force-pushed the aitools/typed-install-errors branch from 4deb804 to 43ac795 Compare November 12, 2026 12:37
@anton-107
anton-107 force-pushed the aitools/windows-shim-probe branch from b28c0c8 to c3363cb Compare November 12, 2026 12:37
@anton-107
anton-107 force-pushed the aitools/typed-install-errors branch from 43ac795 to 9ef9292 Compare November 12, 2026 16:38
@anton-107
anton-107 force-pushed the aitools/windows-shim-probe branch from c3363cb to 3daa09d Compare November 12, 2026 16:38
@anton-107
anton-107 force-pushed the aitools/typed-install-errors branch from 9ef9292 to 6bd9077 Compare November 12, 2026 17:13
@anton-107
anton-107 force-pushed the aitools/windows-shim-probe branch from 3daa09d to 7342f80 Compare November 12, 2026 17:13
@anton-107
anton-107 force-pushed the aitools/typed-install-errors branch from 6bd9077 to 3f72232 Compare November 12, 2026 20:53
@anton-107
anton-107 changed the base branch from aitools/typed-install-errors to main November 12, 2026 21:20
…under its own reason

On Windows, the `codex.cmd` and `copilot.bat` shims can print the full `plugin --help` and then exit 1. Because of that, `databricks aitools install` reported a CLI that is present as `cli-not-on-path`, which points the user at the wrong fix. The probe now reports that case as `shim-exited-with-error` and still stops before running any plugin command:

```diff
 probePluginCLI(agent)
   bin = lookPath(agent CLI)
   err = run `<bin> plugin --help` (20s timeout)
-  if err: block cli-not-on-path
+  if err and windowsShimPrintedHelp(bin, err): block shim-exited-with-error
+      ("printed its plugin help but exited with an error (<exit>)[: <stderr>];
+        to see the error and exit code, run
+        `& "<bin>" plugin --help; $LASTEXITCODE` in PowerShell or
+        `"<bin>" plugin --help` and then `echo %ERRORLEVEL%` in Command Prompt, and
+        re-run this command once it exits with 0")
+  if err: block cli-not-on-path

+windowsShimPrintedHelp(bin, err)
+  bin ends in .cmd or .bat (LookPath returns these only on Windows)
+  and err is not a timeout
+  and stdout has a `Usage: <bin> plugin` line
+      and indented `install`/`add` and `marketplace` command lines
```

`blockedErrorCategory` maps the new reason to `PLUGIN_INSTALL_FAILED`, the closest existing category.

Changelog fragment: `.nextchanges/cli/aitools-windows-shim-probe.md`.

Co-authored-by: Isaac <no-reply@databricks.com>
@anton-107
anton-107 force-pushed the aitools/windows-shim-probe branch from 7342f80 to 2326b6b Compare November 12, 2026 21:29
@anton-107
anton-107 added this pull request to the merge queue Nov 12, 2026
Merged via the queue into main with commit 893aaa3 Nov 12, 2026
24 checks passed
@anton-107
anton-107 deleted the aitools/windows-shim-probe branch November 12, 2026 21:43
idelafuente-oc pushed a commit to idelafuente-oc/cli that referenced this pull request Nov 14, 2026
Stacked on databricks#7476.

Stacked on: aitools: accept a Windows shim probe that printed help but
exited non-zero

## Summary

Coding agents often run commands in a pseudo-terminal (Cursor, Copilot
in the VS Code terminal). Stdin looks like a TTY, so `databricks aitools
install` showed its scope select and agent picker, and nobody was there
to answer them. The run blocked.

Now a caller that `useragent.AgentProvider()` identifies as an agent
always gets the non-interactive defaults: global scope and every
detected agent.

On a terminal, when a prompt would otherwise have run, it prints a note
to stderr that names the detected agent by its `--agents` value (or says
more than one was detected) and the missing `--scope`/`--agents` flags
of `databricks aitools install`, for example `No prompts shown because a
coding agent (cursor) was detected. Pass --scope and --agents to
databricks aitools install to choose what to install.` An agent env var
inherited from an agent session can reach a person's shell, and the note
tells them why the picker is gone.

The change is scoped to install. Follow-up scope, not in this PR: these
prompts still check only `cmdio.IsPromptSupported`, so an agent in a
pseudo-terminal can still block on them.

- the `Proceed?` confirm of `aitools uninstall`
- the both-scopes prompts of `aitools update` and `aitools uninstall`
- the `apps init` skills prompt

```diff
 install RunE
+  interactive := cmdio.IsPromptSupported(ctx) && useragent.AgentProvider() == ""
-  resolveScopeWithPrompt(ctx)                         // prompted on any TTY
+  resolveScopeWithPrompt(ctx, interactive)
-  selectAgents(ctx, scope, skillsOnly)                // picker on any TTY
+  selectAgents(ctx, scope, skillsOnly, interactive)
-  if !explicit && cmdio.IsPromptSupported(ctx)        // plan summary + confirm
+  if !explicit && interactive
```

- The `install` help text now describes the non-interactive defaults.
- The install funnel records `interactive` from the same check, so an
agent caller's run is reported as non-interactive.
- The interactive install tests clear the agent env with
`isolateAgents`, so they keep prompting when the test binary runs under
an agent (`CLAUDECODE`, `AI_AGENT`, ...). The agent-caller test sets
`CURSOR_AGENT=1` and fails if the scope prompt, the picker or the plan
confirm runs. It also checks that the note is printed on stderr. Further
tests check that the note lists only the missing flags, shows the
`--agents` value for the detected agent (`copilot` for `COPILOT_CLI`, no
name for `vscode-agent`, "more than one" for nested agents), and stays
silent when both flags are passed or there is no terminal.
- Changelog fragment:
`.nextchanges/cli/aitools-install-no-prompts-for-agents.md`.

## Evidence

`TestInstallAgentCallerInPTYNeverPrompts`: prompts are supported, the
caller is an agent, and the scope and picker prompts fail the test if
they are called.

- **Before** (`installPromptsSupported` reverted locally to
`cmdio.IsPromptSupported(ctx)`):
  ```
  --- FAIL: TestInstallAgentCallerInPTYNeverPrompts (0.00s)
      install_test.go:887: agent picker shown to an agent caller
  ```
  **After** (this branch):
  ```
  ok  	github.com/databricks/cli/cmd/aitools	0.195s
  ```

Build, unit and acceptance tests for the aitools packages, lint, fmt and
`go vet` for GOOS=windows/darwin pass on this branch. Nothing was run on
a real Windows or macOS machine.

## Merge Danger

**Door:** two-way

Reverting is one small commit. Nothing persistent changes.

**Blast Radius:** small

- Behavior change: when an agent env var is set (for example
`CLAUDECODE`, `AI_AGENT`), a person typing `databricks aitools install`
in a terminal no longer sees prompts. The run installs for every
detected agent in global scope and prints a note explaining why. To
pick, pass `--scope`/`--agents`.
- Exit codes are unchanged, and there are no new network calls.
- Agent detection comes from the SDK's `useragent.AgentProvider()`.
Agents it does not recognize still get prompts on a TTY.

This pull request and its description were written by Isaac.

Co-authored-by: Isaac <no-reply@databricks.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ai-auto PRs made by AI loops & factories,

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants