Repository navigation
aitools: Hint the AI Runtime skill only on air subcommands - #7622
Draft
artemy-kolesnikov wants to merge 3 commits into
Draft
artemy-kolesnikov wants to merge 3 commits into
artemy-kolesnikov wants to merge 3 commits into
Conversation
The Claude Code install hint fired on every non-aitools command with a
generic line ("Databricks skills are not installed..."). Coding agents
treat it as unrelated noise on commands like `auth profiles`, and it
never named the skill that helps with the task at hand.
MaybeHint now fires only on runnable `databricks air` subcommands and
names the databricks-ai-runtime skill, what it covers, the install
command, and a no-install preview (`aitools install --path`). Detection
looks for that skill by exact name in the home stores and the project
store (cwd up to the repository root), and counts the databricks plugin
only when it applies to cwd. Bare `air` and `air --help` render help
without the pre-run hook, so the air help footer now carries the same
benefit line and the preview command. Telemetry keeps INSTALL_TEXT and
the per-session throttle is unchanged.
Co-authored-by: Isaac <no-reply@databricks.com>
Contributor
|
Commit: Regression test reportSelected 5 changed tests (limit=20, 0 not selected) 5 tests: regression=5 unreleased=0 coverage=0 failing=0
TestAccept/air/help: branch ✅ | main (c4559ed) ❌ | latest (v1.26.0) ❌main (c4559ed): latest (v1.26.0): TestAccept/cmd/aitools/claude-config-dir: branch ✅ | main (c4559ed) ❌ | latest (v1.26.0) ❌main (c4559ed): latest (v1.26.0): same output as main (c4559ed) TestAccept/cmd/aitools/telemetry/apps-init-tip: branch ✅ | main (c4559ed) ❌ | latest (v1.26.0) ❌main (c4559ed): latest (v1.26.0): TestAccept/cmd/aitools/telemetry/hint: branch ✅ | main (c4559ed) ❌ | latest (v1.26.0) ❌main (c4559ed): latest (v1.26.0): TestAccept/cmd/version: branch ✅ | main (c4559ed) ❌ | latest (v1.26.0) ❌main (c4559ed): latest (v1.26.0): |
Collaborator
✅ Integration test reportCommit: f38ee96
Top 3 slowest tests (at least 2 minutes):
|
Co-authored-by: Isaac <no-reply@databricks.com>
Windows paths contain backslashes, so interpolating them into the manifest string produced invalid JSON escapes and the plugin record was silently dropped, hinting when the test expected silence. Marshal the manifest instead. Co-authored-by: Isaac <no-reply@databricks.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Title: aitools: Hint the AI Runtime skill only on air subcommands
Changes
The Claude Code install hint (
agents.MaybeHint, run from the rootPersistentPreRunEsince #5918) printed one generic line on every non-aitoolscommand:This change limits the hint to runnable
databricks airsubcommands (run,get,list,logs,cancel,convert-to-dabs,images/images push,pools/pools list/pools get). There it names the skill, says what it covers, and gives both the install and a no-install preview:auth profiles,version,jobs listandapps init(apps initstill prints its ownAPPS_INIT_TIP).airandair --help: cobra renders help without runningPersistentPreRunE, so the hook can't fire there. Theairhelp footer now has the benefit line and the--pathpreview under the existing "Coding agent skill" line.databricks-ai-runtimedir (symlinks followed) in~/.databricks/aitools/skills, the legacy~/.databricks/agent-skills, or the project store<dir>/.databricks/aitools/skills, where<dir>is cwd or any parent up to the repository root (projectDirs, the same walkDatabricksToolinguses). Or the databricks plugin, counted only when its record applies to cwd (databricksPluginActiveIn); a plugin installed for another project no longer silences the hint. Before this change, anydatabricks*skill (for example onlydatabricks-jobs) counted as installed.HasDatabricksSkillsInand its other callers (apps init,version --check) are unchanged.CLAUDE_CODE_SESSION_IDthrottle (at most once per hour per session; with no session id the hint prints on every air command, as before), and the telemetry kind (INSTALL_TEXT). No new enum value;ExecutionContext.Commandalready tells air commands apart..nextchanges/cli/air-skill-hint.mdis added.Why
In a coding-agent run (Claude Code), the agent saw the generic hint on
databricks auth profiles. It read the hint as unrelated setup noise and went to docs instead of the skill. This change keeps the hint to relevant (air) commands, naming the skill and a concrete benefit, with a no-install preview. #7452's description makes the same point: calls with no task to apply the skills to "do not convert".Needs maintainer sign-off on the hint wording. The generic hint is an existing feature, and removing it from non-air commands is a product call. If a generic hint should stay somewhere, the AIR branch of this change can be kept and the generic one restored for those commands, with a separate throttle key per hint kind. Otherwise a generic hint on an earlier command uses up the session's one hint before the agent reaches
air.Why this is based on
main, not stacked on #7452 or #7578:auth,configure,versionand the rest. The two PRs touch the same lines inhint.go, so whichever lands second needs a small rebase.InteractiveModeFull, which needs a TTY. Coding agents run the CLI without one. We checked under the Claude Code Bash tool: stdin, stdout and stderr are all non-TTY,CLAUDECODE=1, andCLAUDE_CODE_SESSION_IDis set. So Suppress AI tooling hints in non-interactive sessions #7578 as written would also hide this hint from the agents it is meant for.TestMaybeHint_SessionThrottleruns with non-TTY stderr and expects the hint, which pins that divergence. If Suppress AI tooling hints in non-interactive sessions #7578 lands, the air hint needs to be exempt from its TTY gate.Tests
New or changed unit tests:
libs/aitools/agents/hint_test.go:TestMaybeHint_AIRSubcommandsHint: runnable air subcommands, including nested ones.TestMaybeHint_NonAIRCommandsSilent:auth profiles,current-user me,clusters list,jobs list,bundle deploy,workspace list,version,aitools install,experimental aitools install, bareair, hiddenexperimental air.TestMaybeHint_AIRSkillDetectionandTestMaybeHint_AIRSkillInRepositoryRoot: canonical, legacy, symlinked and project stores; repository root seen from a subdirectory; otherdatabricks*skills only; a file named like the skill; a plugin for this project vs. an unrelated project.TestMaybeHint_SessionThrottle: with a session id, only the first air command is hinted, and an earlier non-air command doesn't use up the hint. With no session id, every air command is hinted.TestMaybeHint_OtherSessionStillHinted,TestAIRHintText.INSTALL_TEXTevent, or none.cmd/air/hint_test.go:TestRunnableAirCommandsGetSkillHintwalks the realairtree, so a new subcommand is covered automatically.TestBareAirAndHelpShowFooterWithoutPreRunruns bareairandair --help, and asserts the footer content and that the pre-run hook did not run.cmd/air/air_test.go:TestHelpFooterOnlyOnAirCommandalso asserts the skill line, the benefit line and the--pathpreview.Existing tests changed on purpose (they pinned the old policy):
hint_test.go: theregularCmd()stub (a parentlesscurrent-user) is replaced bycommandAt("air list"), since non-air commands no longer hint.TestMaybeHint_SkillsInstalledSilent,_LegacySkillsInstalledSilentand_NonDatabricksSkillStillHintsare folded into the detection table. Their premise, that anydatabricks*skill silences the hint, is the bug._AIToolsCommandSilentis now a row in the non-air table.TestClaudeToolingInstalled_LogsPluginVersionbecameTestAIRSkillAvailable_LogsPluginDetection.claudeToolingInstalledandisAIToolsCommandhad no other callers and were removed.acceptance/cmd/aitools/claude-config-dir: used$CLI versionto exerciseCLAUDE_CONFIG_DIRplugin detection.versionno longer hints, so it uses$CLI air listagainst the mock server. The four expected outcomes are unchanged.acceptance/cmd/aitools/telemetry/hint: the authenticated case isair list(eventair_list/INSTALL_TEXT).jobs listandversionnow show no hint and no event. A new case runsaitools install --path, shows it writes no~/.databricks/aitoolsstate, and shows the next air command is still hinted.acceptance/cmd/aitools/telemetry/apps-init-tip:apps initno longer prints the generic line or logs itsINSTALL_TEXTevent. Its ownAPPS_INIT_TIPis unchanged.acceptance/cmd/version:version --checkfrom Claude Code printed the generic hint. Now it prints nothing.MaybeVersionCheckTipstill skips Claude Code, so Claude gets no tip there; tell us if you'd rather it get the tip like other agents.acceptance/air/help: golden updated for the footer.Results:
go test ./libs/aitools/... ./cmd/air/... ./cmd/root/... ./cmd/version/... ./libs/telemetry/...passes. AcceptanceTestAccept/cmd/aitools,TestAccept/cmd/version,TestAccept/air,TestAccept/cmd/appsandTestAccept/appspass.golangci-linton the touched packages reports 0 issues, andgofmt,validate_nextchanges.pyandvalidate_whitespace.pyare clean.Manual end-to-end check: a local build in a fresh temp HOME, no workspace calls, run under the Claude Code Bash tool (non-TTY):
CLAUDECODE=1, no session id)mainstderrauth profilesversionair listair --help/ bareair--pathpreviewaitools install --path $HOME/preview --skills databricks-ai-runtimeSKILL.md, no stateair liststill hintedair listwith onlydatabricks-jobsinstalledair listwithdatabricks-ai-runtimein<cwd>/.databricks/aitools/skillsair listwith the plugin installed for another projectair list, thenair get 1, same session idair list/auth profileswithoutCLAUDECODEThis pull request and its description were written by Isaac.