Repository navigation
aitools: report a Windows shim that printed help but exited non-zero under its own reason - #7476
Conversation
9132948 to
405e077
Compare
5601cd2 to
2c28258
Compare
✅ Integration test reportCommit: b28c0c8
Top 34 slowest tests (at least 2 minutes):
|
405e077 to
1aa2b9f
Compare
2c28258 to
4a70459
Compare
1aa2b9f to
bbd0a95
Compare
4a70459 to
c162e8e
Compare
bbd0a95 to
f18c292
Compare
c162e8e to
0c0d208
Compare
simonfaltum
left a comment
There was a problem hiding this comment.
🤖 Automated review from simonfaltum's review agent (pr-review skill) at 0c0d208. Each finding was verified against the code; simonfaltum may follow up.
Not approving yet: 1 blocking issue(s), tagged [blocking]; 1 open design question.
This PR makes the plugin probe accept a Windows .cmd/.bat agent CLI that prints its plugin --help and then exits non-zero, instead of reporting it as cli-not-on-path. The gate is narrow and the new tests pass. Two things should be resolved before merge. First, the three help regexes are compiled in package-level vars, which the repo's Go style guide forbids. Second, the fix rests on a cause nobody has verified: if the shim exits non-zero on every command, install now fails at a later step and leaves a marketplace and plugin that aitools doesn't track. The rest are test-coverage, comment and changelog fixes.
Questions
- Do
codex.cmdandcopilot.batexit non-zero only onplugin --help, or on every command that runs through the shim? The change hasn't run on Windows, and the tests use a hand-written help string. If every call exits non-zero, accepting the probe meansmarketplace addandplugin installnow run, reportinstall-failed, and leave a marketplace and plugin that aitools doesn't track. Today the probe stops before anything changes. Could you capture the real stdout, stderr and exit codes on Windows forplugin --helpand for a plugin install through each shim, and use that output as the test fixture? If the shim layer turns out to be the cause, would it fit better to handle it inrunAgentCmd, so install, update and uninstall all benefit? Another option is to report this case under its own reason instead ofcli-not-on-path.
Comments outside the changed lines
- [nit]
.github/PULL_REQUEST_TEMPLATE.md: The description uses## Summary,## Evidenceand## Merge Danger..agents/skills/pr-checklist/SKILL.mdasks agent-written PRs to use the template's## Changes,## Whyand## Tests, in that order. It names## Summaryas a section not to add, and asks for the agent disclosure as a single italic last line. Move the content under the template headings: Evidence fits under Tests, and Merge Danger under Why.
| pluginHelpUsage = regexp.MustCompile(`(?im)^usage:[ \t]+\S+[ \t]+plugin\b`) | ||
| pluginHelpInstall = regexp.MustCompile(`(?m)^[ \t]+(install|add)\b`) | ||
| pluginHelpMarket = regexp.MustCompile(`(?m)^[ \t]+marketplace\b`) |
There was a problem hiding this comment.
[blocking] These three regexp.MustCompile calls run in package-level var initializers, which .agents/rules/style-guide-go.md forbids ("Do not compile regexps ... in package-level var initializers or init()"). The main binary links libs/aitools/installer, so every databricks command on every OS compiles three patterns. Only the Windows .cmd/.bat path uses them, after a failed probe. Wrap each one in sync.OnceValue(func() *regexp.Regexp { return regexp.MustCompile(...) }) as cmd/air/runconfig.go:31 does. Then call them only after the GOOS, extension and timeout checks in windowsShimPrintedHelp. The new tests already go through that function, so a bad pattern still panics in tests.
There was a problem hiding this comment.
Fixed in e158591: the three patterns are now sync.OnceValue funcs (same shape as cmd/air/runconfig.go), and windowsShimPrintedHelp calls them only after the GOOS, extension, timeout and ProcessError checks. The table test exercises all three, so a bad pattern still panics in tests.
| accept bool | ||
| }{ | ||
| {name: "windows .bat shim printed help and exited 1", goos: "windows", bin: `C:\npm\copilot.bat`, stdout: copilotPluginHelp, accept: true}, | ||
| {name: "windows .cmd shim printed help and exited 1", goos: "windows", bin: `C:\npm\codex.cmd`, stdout: copilotPluginHelp, accept: true}, |
There was a problem hiding this comment.
[nice to have] The .cmd case uses a codex.cmd path but gives the Copilot agent Copilot's commander-style help, which lists install. Codex prints clap-style help (Usage: codex plugin [OPTIONS] <COMMAND>) with an add subcommand. So the add branch of pluginHelpInstall, which exists only for Codex, is never tested. If that branch or the Codex help shape broke, every test would still pass, and codex.cmd would go back to reporting cli-not-on-path. Add a row with codexAgent() and Codex's real plugin --help output, ideally captured on Windows.
There was a problem hiding this comment.
Added in e158591: a codexAgent() row with codex.cmd and clap-style help (Usage: codex plugin [OPTIONS] <COMMAND> listing add and marketplace), so the add branch is covered. The former .cmd row now uses copilot.cmd, so it no longer mixes Codex's path with Copilot's help. The Codex fixture is a representative shape, not output captured on Windows; that capture belongs to the open Question 1.
| // probeGOOS is runtime.GOOS; tests replace it to exercise the Windows path. | ||
| var probeGOOS = runtime.GOOS | ||
|
|
||
| // pluginHelpUsage and pluginHelpCommand match the help that `<agent> plugin |
There was a problem hiding this comment.
[nit] The comment says "pluginHelpUsage and pluginHelpCommand match…", but there is no pluginHelpCommand; the variables are pluginHelpUsage, pluginHelpInstall and pluginHelpMarket. In the windowsShimPrintedHelp doc below, line 290 runs to about 92 columns while the lines around it wrap near 80. That doc's second half also lists the guard clauses the code already shows. Fix the name, rewrap the line, and keep only the reason for the quirk.
There was a problem hiding this comment.
Fixed in e158591: the comment names pluginHelpUsage, pluginHelpInstall and pluginHelpMarket (noting add for Codex), and the windowsShimPrintedHelp doc is rewrapped and keeps only the reason for the quirk.
| assert.False(t, windowsShimPrintedHelp(bin, &agentCmdTimeoutError{timeout: time.Second, err: perr})) | ||
| } | ||
|
|
||
| func TestProbeRejectsCLIWithoutPluginCommand(t *testing.T) { |
There was a problem hiding this comment.
[nit] This test leaves probeGOOS at the host value and uses stubAgentLookPath, which returns /usr/bin/codex with no extension. So windowsShimPrintedHelp returns false at the GOOS or extension check, before any regex runs, and the test passes on the base commit too. Make it a row in the TestProbeWindowsShimPrintedHelp table with goos: "windows", a .cmd bin and the unknown command 'plugin' output. Then it actually checks that the matcher rejects an error.
There was a problem hiding this comment.
Fixed in e158591: TestProbeRejectsCLIWithoutPluginCommand is gone; it is now the windows shim without a plugin command row in TestProbeWindowsShimPrintedHelp (goos: "windows", C:\npm\codex.cmd, unknown command 'plugin' output), so it gets past the GOOS and extension checks and tests the matcher.
| return "", &BlockedError{Agent: agent.Name, Reason: ReasonCLINotOnPath, Detail: err.Error()} | ||
| } | ||
| if _, err := runAgentCmd(ctx, pluginProbeTimeout, []string{bin, "plugin", "--help"}); err != nil { | ||
| if _, err := runAgentCmd(ctx, pluginProbeTimeout, []string{bin, "plugin", "--help"}); err != nil && !windowsShimPrintedHelp(bin, err) { |
There was a problem hiding this comment.
[nit] When windowsShimPrintedHelp accepts a failed probe, the error is dropped without a log. If a later step fails, debug logs show nothing about the probe's non-zero exit, so Windows reports are hard to diagnose. Log it at debug level in probePluginCLI, where ctx is available, e.g. log.Debugf(ctx, "accepting %s plugin --help that exited non-zero after printing help: %v", bin, err).
There was a problem hiding this comment.
Fixed in e158591: probePluginCLI now logs accepting %s plugin --help that exited non-zero after printing help: %v at debug level when it accepts the failed probe.
There was a problem hiding this comment.
Update in f6b4ec8: following the decision on Question 1, a shim that printed the help and exited non-zero is now blocked as shim-exited-with-error instead of accepted. So the debug log is gone; the exit status is in the error's detail that the user sees.
| @@ -0,0 +1 @@ | |||
| * `databricks aitools install` on Windows accepts a coding agent CLI shim (`.cmd` or `.bat`) that prints its plugin help and then exits with an error, instead of reporting the CLI as not on PATH. ([#7476](https://github.com/databricks/cli/pull/7476)) | |||
There was a problem hiding this comment.
[nit] The entry names only databricks aitools install, but aitools update reaches the same probe through its legacy raw-skills migration (cmd/aitools/update.go:253 calls InstallPluginForAgent). The earlier probe-timeout fragment listed both commands. pr-checklist also asks a command-led entry to say "now". Suggest: "* databricks aitools install and databricks aitools update now accept a coding agent CLI shim (.cmd or .bat) on Windows that prints its plugin help and then exits with an error, instead of reporting the CLI as not on PATH. (#7476)"
There was a problem hiding this comment.
Fixed in e158591: the fragment now uses your wording, naming both databricks aitools install and databricks aitools update and keeping the PR link.
f18c292 to
af32385
Compare
0c0d208 to
873e0e0
Compare
af32385 to
2408150
Compare
eabdd30 to
e158591
Compare
|
Thanks. Pushed e158591: the [blocking] regexps are now On the PR description: I kept the current sections. This stack uses the same Summary / Evidence / Merge Danger layout on every PR, so reviewers can read them all the same way. The content covers what the template asks for (what changed, why, and how it was tested). Question 1 (real Windows shim behavior and where to handle it) is still open. I'll follow up on it separately. |
e158591 to
f6b4ec8
Compare
|
On Question 1: we couldn't verify how
Since the install no longer goes ahead, this doesn't touch |
2408150 to
5275455
Compare
f6b4ec8 to
3108ec5
Compare
2d5e5ca to
64a25aa
Compare
6731eb1 to
a7cd2a4
Compare
64a25aa to
7f0dcdf
Compare
a7cd2a4 to
d0cb9df
Compare
simonfaltum
left a comment
There was a problem hiding this comment.
🤖 Approval from simonfaltum's review agent at d0cb9df, on Simon's decision.
The design questions from the earlier automated reviews no longer block this PR: your answers are accepted, and any still-open ones are non-blocking. The changes since the last automated review (149851e → d0cb9df) address its comments.
rugpanov
left a comment
There was a problem hiding this comment.
Also found by our reviewers, already asked in simonfaltum's review (Question 1): the new reason fires only for a .cmd or .bat CLI whose help matches the three regexes. Our first-round notes asked the same: a split with no help parsing (the CLI was found, but plugin --help failed) gives the same signal on every platform.
| if stderr := strings.TrimSpace(perr.Stderr); stderr != "" { | ||
| detail += ": " + stderr | ||
| } | ||
| return detail + fmt.Sprintf("; to see the error and exit code, run `& \"%[1]s\" plugin --help; $LASTEXITCODE` in PowerShell or `\"%[1]s\" plugin --help & echo %%ERRORLEVEL%%` in Command Prompt, and re-run this command once it exits with 0", bin) |
There was a problem hiding this comment.
[nice to have] In the Command Prompt form, "<bin>" plugin --help & echo %ERRORLEVEL%, cmd.exe expands %ERRORLEVEL% when it reads the line, before the shim runs. So it prints the exit code of the previous command, usually 0. The hint then says to re-run "once it exits with 0", so the user can think that the shim is fixed when it is not. Tell the user to run echo %ERRORLEVEL% as a separate command, and update the expected text in TestProbeWindowsShimPrintedHelp.
There was a problem hiding this comment.
Fixed in 735616f. The Command Prompt form now has two separate commands: "<bin>" plugin --help, then echo %ERRORLEVEL%. The second one reads the shim's exit code after it ran, not the code from when the line was read. The shimExitedDetail doc says why, and the expected text in TestProbeWindowsShimPrintedHelp is updated. The PowerShell form (& "<bin>" plugin --help; $LASTEXITCODE) stays as it was, since PowerShell evaluates $LASTEXITCODE after the call.
| // the agent CLIs print --help to stdout, and help on stderr usually means the | ||
| // CLI rejected its arguments. | ||
| // | ||
| // No public upstream report documents this exit code, so there is no link to |
There was a problem hiding this comment.
[nit] "No public upstream report documents this exit code, so there is no link to cite" answers a review question. It does not tell the caller anything. .agents/rules/comments.md asks for the contract only. Keep the pointer to TestProbeRealCmdShimPrintedHelp, and remove the rest of this paragraph.
There was a problem hiding this comment.
7f0dcdf to
45cb4be
Compare
1988e05 to
735616f
Compare
45cb4be to
dccb816
Compare
735616f to
e26c489
Compare
|
Thanks. On the generic probe-failure reason: that's Question 1 from simonfaltum's review, answered in #7476 (comment). This PR stays limited to Windows Also in 735616f: the |
dccb816 to
4deb804
Compare
e26c489 to
b28c0c8
Compare
4deb804 to
43ac795
Compare
b28c0c8 to
c3363cb
Compare
43ac795 to
9ef9292
Compare
c3363cb to
3daa09d
Compare
9ef9292 to
6bd9077
Compare
3daa09d to
7342f80
Compare
6bd9077 to
3f72232
Compare
…under its own reason
On Windows, the `codex.cmd` and `copilot.bat` shims can print the full `plugin --help` and then exit 1. Because of that, `databricks aitools install` reported a CLI that is present as `cli-not-on-path`, which points the user at the wrong fix. The probe now reports that case as `shim-exited-with-error` and still stops before running any plugin command:
```diff
probePluginCLI(agent)
bin = lookPath(agent CLI)
err = run `<bin> plugin --help` (20s timeout)
- if err: block cli-not-on-path
+ if err and windowsShimPrintedHelp(bin, err): block shim-exited-with-error
+ ("printed its plugin help but exited with an error (<exit>)[: <stderr>];
+ to see the error and exit code, run
+ `& "<bin>" plugin --help; $LASTEXITCODE` in PowerShell or
+ `"<bin>" plugin --help` and then `echo %ERRORLEVEL%` in Command Prompt, and
+ re-run this command once it exits with 0")
+ if err: block cli-not-on-path
+windowsShimPrintedHelp(bin, err)
+ bin ends in .cmd or .bat (LookPath returns these only on Windows)
+ and err is not a timeout
+ and stdout has a `Usage: <bin> plugin` line
+ and indented `install`/`add` and `marketplace` command lines
```
`blockedErrorCategory` maps the new reason to `PLUGIN_INSTALL_FAILED`, the closest existing category.
Changelog fragment: `.nextchanges/cli/aitools-windows-shim-probe.md`.
Co-authored-by: Isaac <no-reply@databricks.com>
7342f80 to
2326b6b
Compare
Stacked on databricks#7476. Stacked on: aitools: accept a Windows shim probe that printed help but exited non-zero ## Summary Coding agents often run commands in a pseudo-terminal (Cursor, Copilot in the VS Code terminal). Stdin looks like a TTY, so `databricks aitools install` showed its scope select and agent picker, and nobody was there to answer them. The run blocked. Now a caller that `useragent.AgentProvider()` identifies as an agent always gets the non-interactive defaults: global scope and every detected agent. On a terminal, when a prompt would otherwise have run, it prints a note to stderr that names the detected agent by its `--agents` value (or says more than one was detected) and the missing `--scope`/`--agents` flags of `databricks aitools install`, for example `No prompts shown because a coding agent (cursor) was detected. Pass --scope and --agents to databricks aitools install to choose what to install.` An agent env var inherited from an agent session can reach a person's shell, and the note tells them why the picker is gone. The change is scoped to install. Follow-up scope, not in this PR: these prompts still check only `cmdio.IsPromptSupported`, so an agent in a pseudo-terminal can still block on them. - the `Proceed?` confirm of `aitools uninstall` - the both-scopes prompts of `aitools update` and `aitools uninstall` - the `apps init` skills prompt ```diff install RunE + interactive := cmdio.IsPromptSupported(ctx) && useragent.AgentProvider() == "" - resolveScopeWithPrompt(ctx) // prompted on any TTY + resolveScopeWithPrompt(ctx, interactive) - selectAgents(ctx, scope, skillsOnly) // picker on any TTY + selectAgents(ctx, scope, skillsOnly, interactive) - if !explicit && cmdio.IsPromptSupported(ctx) // plan summary + confirm + if !explicit && interactive ``` - The `install` help text now describes the non-interactive defaults. - The install funnel records `interactive` from the same check, so an agent caller's run is reported as non-interactive. - The interactive install tests clear the agent env with `isolateAgents`, so they keep prompting when the test binary runs under an agent (`CLAUDECODE`, `AI_AGENT`, ...). The agent-caller test sets `CURSOR_AGENT=1` and fails if the scope prompt, the picker or the plan confirm runs. It also checks that the note is printed on stderr. Further tests check that the note lists only the missing flags, shows the `--agents` value for the detected agent (`copilot` for `COPILOT_CLI`, no name for `vscode-agent`, "more than one" for nested agents), and stays silent when both flags are passed or there is no terminal. - Changelog fragment: `.nextchanges/cli/aitools-install-no-prompts-for-agents.md`. ## Evidence `TestInstallAgentCallerInPTYNeverPrompts`: prompts are supported, the caller is an agent, and the scope and picker prompts fail the test if they are called. - **Before** (`installPromptsSupported` reverted locally to `cmdio.IsPromptSupported(ctx)`): ``` --- FAIL: TestInstallAgentCallerInPTYNeverPrompts (0.00s) install_test.go:887: agent picker shown to an agent caller ``` **After** (this branch): ``` ok github.com/databricks/cli/cmd/aitools 0.195s ``` Build, unit and acceptance tests for the aitools packages, lint, fmt and `go vet` for GOOS=windows/darwin pass on this branch. Nothing was run on a real Windows or macOS machine. ## Merge Danger **Door:** two-way Reverting is one small commit. Nothing persistent changes. **Blast Radius:** small - Behavior change: when an agent env var is set (for example `CLAUDECODE`, `AI_AGENT`), a person typing `databricks aitools install` in a terminal no longer sees prompts. The run installs for every detected agent in global scope and prints a note explaining why. To pick, pass `--scope`/`--agents`. - Exit codes are unchanged, and there are no new network calls. - Agent detection comes from the SDK's `useragent.AgentProvider()`. Agents it does not recognize still get prompts on a TTY. This pull request and its description were written by Isaac. Co-authored-by: Isaac <no-reply@databricks.com>
Stacked on #7475.
Stacked on: aitools: typed fetch, state and agent CLI timeout errors
Summary
On Windows, the
codex.cmdandcopilot.batshims can print the fullplugin --helpand then exit 1. Because of that,databricks aitools installreported a CLI that is present ascli-not-on-path, which points the user at the wrong fix. The probe now reports that case asshim-exited-with-errorand still stops before running any plugin command:blockedErrorCategorymaps the new reason toPLUGIN_INSTALL_FAILED, the closest existing category.Changelog fragment:
.nextchanges/cli/aitools-windows-shim-probe.md.Evidence
Tests in
libs/aitools/installer/plugin_test.go. The stubbedlookPathreturns.cmd/.batpaths, so the Windows path runs on any platform.plugin_windows_test.goalso runs a real.cmdthat prints the help and exits 1, on the Windows CI cells only.probePluginCLI, otherwise this head):Merge Danger
Door: two-way
Only a code path changes. No state or config changes. In
--output json, this case now reportserror_categoryPLUGIN_INSTALL_FAILEDinstead ofCLI_NOT_ON_PATH.Blast Radius: narrow
.cmd/.batagent CLI whoseplugin --helpexits non-zero but prints the plugin help is now blocked asshim-exited-with-errorinstead ofcli-not-on-path, with the command to run to see the error. As before, no marketplace or plugin command runs. On every other platform, for.exe, and for timeouts, the probe behaves as before.cli-not-on-path.This pull request and its description were written by Isaac.