Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .nextchanges/cli/air-skill-hint.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
* Claude Code gets the Databricks skills install hint only on `databricks air` subcommands, where it names the `databricks-ai-runtime` skill, what it covers, and how to preview it with `databricks aitools install --path`; other commands no longer print the hint, and `databricks air --help` lists the preview too. ([#7622](https://github.com/databricks/cli/pull/7622))
4 changes: 4 additions & 0 deletions acceptance/air/help/output.txt
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,10 @@ Learn more:
Quickstart (AWS) https://docs.databricks.com/aws/en/machine-learning/ai-runtime/quickstart
Quickstart (Azure) https://learn.microsoft.com/azure/databricks/machine-learning/ai-runtime/quickstart
Coding agent skill databricks aitools install --skills-only --skills databricks-ai-runtime
Teaches coding agents the workload YAML, GPU types, secrets,
and how to follow and debug runs.
Preview the skill databricks aitools install --path <dir> --skills databricks-ai-runtime
Writes the skill files to <dir> without installing anything.

=== list help

Expand Down
18 changes: 9 additions & 9 deletions acceptance/cmd/aitools/claude-config-dir/output.txt
Original file line number Diff line number Diff line change
@@ -1,21 +1,21 @@

=== Plugin installed only in the custom config directory

>>> CLAUDE_CONFIG_DIR=custom [CLI] version
Databricks CLI v[CLI_VERSION]
>>> CLAUDE_CONFIG_DIR=custom [CLI] air list
No active runs found. Use --all-status to include runs that are no longer active.

=== Custom directory takes precedence over the default installation

>>> CLAUDE_CONFIG_DIR=missing [CLI] version
Databricks skills are not installed. To work with Databricks reliably, first run: databricks aitools install
Databricks CLI v[CLI_VERSION]
>>> CLAUDE_CONFIG_DIR=missing [CLI] air list
Tip: the databricks-ai-runtime skill is not installed. It documents the air workload YAML, GPU types, secrets, and how to follow and debug runs. Install it: databricks aitools install --skills-only --skills databricks-ai-runtime (to read it first without installing: databricks aitools install --path <dir> --skills databricks-ai-runtime)
No active runs found. Use --all-status to include runs that are no longer active.

=== Empty override uses the default installation

>>> CLAUDE_CONFIG_DIR= [CLI] version
Databricks CLI v[CLI_VERSION]
>>> CLAUDE_CONFIG_DIR= [CLI] air list
No active runs found. Use --all-status to include runs that are no longer active.

=== Unset override uses the default installation

>>> [CLI] version
Databricks CLI v[CLI_VERSION]
>>> [CLI] air list
No active runs found. Use --all-status to include runs that are no longer active.
10 changes: 5 additions & 5 deletions acceptance/cmd/aitools/claude-config-dir/script
Original file line number Diff line number Diff line change
@@ -1,18 +1,18 @@
export CLAUDECODE=1
unset CLAUDE_CODE_SESSION_ID

# aitools commands skip the install hint; version exercises it without auth or network.
# Only air subcommands print the install hint; air list against the mock server exercises it.
sethome ./empty-home
title "Plugin installed only in the custom config directory"
trace CLAUDE_CONFIG_DIR=custom $CLI version
trace CLAUDE_CONFIG_DIR=custom $CLI air list

sethome ./installed-home
title "Custom directory takes precedence over the default installation"
trace CLAUDE_CONFIG_DIR=missing $CLI version
trace CLAUDE_CONFIG_DIR=missing $CLI air list

title "Empty override uses the default installation"
trace CLAUDE_CONFIG_DIR= $CLI version
trace CLAUDE_CONFIG_DIR= $CLI air list

title "Unset override uses the default installation"
unset CLAUDE_CONFIG_DIR
trace $CLI version
trace $CLI air list
2 changes: 0 additions & 2 deletions acceptance/cmd/aitools/telemetry/apps-init-tip/output.txt
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,6 @@
=== apps init with flags prints the tip: one hint-shown event per hint printed

>>> [CLI] apps init --template ./template --skip-install --name app-tip
Databricks skills are not installed. To work with Databricks reliably, first run: databricks aitools install
✔ Creating project

✔ Project created successfully!
Expand All @@ -13,7 +12,6 @@ Databricks skills are not installed. To work with Databricks reliably, first run
Tip: coding agents detected without Databricks skills. Run 'databricks aitools install' to install them.

>>> print_telemetry
{"command":"apps_init","exit_code":0,"aitools_hint_event":{"hint_kind":"INSTALL_TEXT"}}
{"command":"apps_init","exit_code":0,"aitools_hint_event":{"hint_kind":"APPS_INIT_TIP"}}

=== plugin installed: no hint, no tip, no event
Expand Down
38 changes: 31 additions & 7 deletions acceptance/cmd/aitools/telemetry/hint/output.txt
Original file line number Diff line number Diff line change
@@ -1,22 +1,46 @@

=== hint printed on an authenticated command: one hint-shown event is uploaded
=== hint printed on an air command: one hint-shown event is uploaded

>>> [CLI] jobs list
Databricks skills are not installed. To work with Databricks reliably, first run: databricks aitools install
>>> [CLI] air list
Tip: the databricks-ai-runtime skill is not installed. It documents the air workload YAML, GPU types, secrets, and how to follow and debug runs. Install it: databricks aitools install --skills-only --skills databricks-ai-runtime (to read it first without installing: databricks aitools install --path <dir> --skills databricks-ai-runtime)
No active runs found. Use --all-status to include runs that are no longer active.

>>> print_telemetry
{"command":"jobs_list","exit_code":0,"aitools_hint_event":{"hint_kind":"INSTALL_TEXT"}}
{"command":"air_list","exit_code":0,"aitools_hint_event":{"hint_kind":"INSTALL_TEXT"}}

=== commands outside air print no hint and log no event

>>> [CLI] jobs list

=== hint printed on a command that resolves no auth: nothing is uploaded
>>> print_telemetry

>>> [CLI] version
Databricks skills are not installed. To work with Databricks reliably, first run: databricks aitools install
Databricks CLI v[CLI_VERSION]

=== previewing a skill with --path installs nothing, so air still hints

>>> [CLI] aitools install --path ./preview --skills databricks-test
Using skills version test-ref
Wrote 1 skill to ./preview

>>> print_telemetry
{"command":"aitools_install","exit_code":0,"aitools_command_event":{"command":"INSTALL","stage":"STARTED","interactive":false,"install_mode":"PATH"}}
{"command":"aitools_install","exit_code":0,"aitools_command_event":{"command":"INSTALL","stage":"STEP_COMPLETED","step":"FLAGS_VALIDATED","interactive":false,"install_mode":"PATH"}}
{"command":"aitools_install","exit_code":0,"aitools_command_event":{"command":"INSTALL","stage":"STEP_COMPLETED","step":"SKILLS_DUMPED","interactive":false,"install_mode":"PATH"}}
{"command":"aitools_install","exit_code":0,"aitools_command_event":{"command":"INSTALL","stage":"FINISHED","outcome":"SUCCESS","interactive":false,"install_mode":"PATH"}}
preview/databricks-test/SKILL.md
no aitools state in HOME

>>> [CLI] air list
Tip: the databricks-ai-runtime skill is not installed. It documents the air workload YAML, GPU types, secrets, and how to follow and debug runs. Install it: databricks aitools install --skills-only --skills databricks-ai-runtime (to read it first without installing: databricks aitools install --path <dir> --skills databricks-ai-runtime)
No active runs found. Use --all-status to include runs that are no longer active.

>>> print_telemetry
{"command":"air_list","exit_code":0,"aitools_hint_event":{"hint_kind":"INSTALL_TEXT"}}

=== plugin installed: no hint, no event

>>> [CLI] jobs list
>>> [CLI] air list
No active runs found. Use --all-status to include runs that are no longer active.

>>> print_telemetry
19 changes: 15 additions & 4 deletions acceptance/cmd/aitools/telemetry/hint/script
Original file line number Diff line number Diff line change
Expand Up @@ -5,15 +5,26 @@ unset CLAUDE_CODE_SESSION_ID
# A host CLAUDE_CONFIG_DIR would move the plugin file away from the isolated HOME.
unset CLAUDE_CONFIG_DIR

title "hint printed on an authenticated command: one hint-shown event is uploaded"
trace $CLI jobs list
title "hint printed on an air command: one hint-shown event is uploaded"
trace $CLI air list
trace print_telemetry

title "hint printed on a command that resolves no auth: nothing is uploaded"
title "commands outside air print no hint and log no event"
trace $CLI jobs list
trace print_telemetry
trace $CLI version

title "previewing a skill with --path installs nothing, so air still hints"
# The mock manifest serves only databricks-test.
trace $CLI aitools install --path ./preview --skills databricks-test
trace print_telemetry
find preview -type f | sort
rm -rf preview
test -e "$HOME/.databricks/aitools" && echo "unexpected aitools state in HOME" || echo "no aitools state in HOME"
trace $CLI air list
trace print_telemetry

title "plugin installed: no hint, no event"
sethome ./installed-home
trace $CLI jobs list
trace $CLI air list
trace print_telemetry
3 changes: 1 addition & 2 deletions acceptance/cmd/version/output.txt
Original file line number Diff line number Diff line change
Expand Up @@ -35,9 +35,8 @@ Tip: coding agents detected without Databricks skills. Run 'databricks aitools i
"development_build": true
}

=== version --check from Claude Code prints only the root hint
=== version --check from Claude Code prints no hint

>>> [CLI] version --check
Databricks skills are not installed. To work with Databricks reliably, first run: databricks aitools install
Databricks CLI v[CLI_VERSION]
This is a development build; skipping the update check.
5 changes: 3 additions & 2 deletions acceptance/cmd/version/script
Original file line number Diff line number Diff line change
Expand Up @@ -23,8 +23,9 @@ trace $CLI version --check
title "version --check --output json with a coding agent prints no tip"
trace $CLI version --check --output json

# Claude Code callers get only the root hint (agents.MaybeHint), not the tip.
title "version --check from Claude Code prints only the root hint"
# Claude Code callers get no tip: agents.MaybeHint owns their hints and only
# hints on air subcommands.
title "version --check from Claude Code prints no hint"
sethome ./claude-home
export CLAUDECODE=1
unset CLAUDE_CODE_SESSION_ID
Expand Down
4 changes: 4 additions & 0 deletions cmd/air/air.go
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,10 @@ Learn more:
Quickstart (AWS) https://docs.databricks.com/aws/en/machine-learning/ai-runtime/quickstart
Quickstart (Azure) https://learn.microsoft.com/azure/databricks/machine-learning/ai-runtime/quickstart
Coding agent skill databricks aitools install --skills-only --skills databricks-ai-runtime
Teaches coding agents the workload YAML, GPU types, secrets,
and how to follow and debug runs.
Preview the skill databricks aitools install --path <dir> --skills databricks-ai-runtime
Writes the skill files to <dir> without installing anything.
`

// New returns the root command for the AI Runtime CLI.
Expand Down
5 changes: 5 additions & 0 deletions cmd/air/air_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,11 @@ func TestHelpFooterOnlyOnAirCommand(t *testing.T) {
cmd.HelpFunc()(cmd, nil)
assert.Contains(t, airOut.String(), "Learn more:")
assert.Contains(t, airOut.String(), "databricks air run -h config")
// The footer is where `air` and `air --help` recommend the agent skill
// (help never reaches the pre-run hook that prints the install hint).
assert.Contains(t, airOut.String(), "databricks aitools install --skills-only --skills databricks-ai-runtime")
assert.Contains(t, airOut.String(), "Teaches coding agents the workload YAML")
assert.Contains(t, airOut.String(), "databricks aitools install --path <dir> --skills databricks-ai-runtime")

list := findSubcommand(t, cmd, "list")
var listOut bytes.Buffer
Expand Down
89 changes: 89 additions & 0 deletions cmd/air/hint_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,89 @@
package aircmd

import (
"bytes"
"testing"

"github.com/databricks/cli/internal/testutil"
"github.com/databricks/cli/libs/aitools/agents"
"github.com/databricks/cli/libs/cmdio"
"github.com/databricks/cli/libs/env"
"github.com/spf13/cobra"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)

// airUnderRoot mounts the real air tree under a root whose pre-run hook only
// records that it ran, mirroring where cmd/root calls agents.MaybeHint.
func airUnderRoot(preRunCalled *bool) (*cobra.Command, *cobra.Command) {
root := &cobra.Command{Use: "databricks"}
root.AddGroup(&cobra.Group{ID: "development", Title: "Developer Tools"})
root.PersistentPreRunE = func(cmd *cobra.Command, args []string) error {
*preRunCalled = true
return nil
}
air := New()
root.AddCommand(air)
return root, air
}

// TestBareAirAndHelpShowFooterWithoutPreRun pins why the footer, not the
// dynamic hint, is the recommendation surface for `air` and `air --help`.
func TestBareAirAndHelpShowFooterWithoutPreRun(t *testing.T) {
for _, args := range [][]string{{"air"}, {"air", "--help"}} {
t.Run(args[len(args)-1], func(t *testing.T) {
preRunCalled := false
root, _ := airUnderRoot(&preRunCalled)
var out bytes.Buffer
root.SetOut(&out)
root.SetErr(&out)
root.SetArgs(args)
require.NoError(t, root.Execute())
assert.False(t, preRunCalled, "help must not run the pre-run hook")
assert.Contains(t, out.String(), "Coding agent skill databricks aitools install --skills-only --skills databricks-ai-runtime")
assert.Contains(t, out.String(), "Preview the skill databricks aitools install --path <dir> --skills databricks-ai-runtime")
})
}
}

// TestRunnableAirCommandsGetSkillHint walks the real air tree, so a new
// subcommand is covered without listing it here.
func TestRunnableAirCommandsGetSkillHint(t *testing.T) {
testutil.SetAgentEnv(t, "CLAUDECODE", "1")
preRunCalled := false
_, air := airUnderRoot(&preRunCalled)
assert.False(t, air.Runnable(), "bare air renders help; the footer covers it")

var runnable []string
var walk func(c *cobra.Command)
walk = func(c *cobra.Command) {
for _, sub := range c.Commands() {
if sub.Runnable() {
runnable = append(runnable, sub.CommandPath())
}
walk(sub)
}
}
walk(air)
for _, name := range []string{"run", "get", "list", "logs", "cancel", "convert-to-dabs"} {
assert.Contains(t, runnable, "databricks air "+name)
}

var visit func(c *cobra.Command)
visit = func(c *cobra.Command) {
for _, sub := range c.Commands() {
if sub.Runnable() {
t.Run(sub.CommandPath(), func(t *testing.T) {
t.Chdir(t.TempDir())
ctx, stderr := cmdio.NewTestContextWithStderr(t.Context())
ctx = env.WithUserHomeDir(ctx, t.TempDir())
ctx = env.Set(ctx, "DATABRICKS_CACHE_DIR", t.TempDir())
agents.MaybeHint(ctx, sub)
assert.Contains(t, stderr.String(), "the databricks-ai-runtime skill is not installed")
})
}
visit(sub)
}
}
visit(air)
}
5 changes: 3 additions & 2 deletions libs/aitools/agents/calling.go
Original file line number Diff line number Diff line change
Expand Up @@ -62,8 +62,9 @@ func (t Tooling) String() string {
// record. For Codex and Copilot, pluginRecorded (the CLI install state) is the
// only plugin source, so it misses plugins installed with the agent's own CLI
// and includes plugins later removed with it. The shared skills store is not consulted: it cannot attribute skills.
// The install hint's claudeToolingInstalled differs on purpose: it stays quiet
// for any trace of an install, while this reports what the agent loads.
// The install hint's airSkillAvailable differs on purpose: it looks for the AI
// Runtime skill only, in the shared store too, while this reports what the
// agent loads.
func (a *Agent) DatabricksTooling(ctx context.Context, pluginRecorded func() bool) Tooling {
// An unknown working directory skips the project lookups below.
cwd := ""
Expand Down
Loading
Loading