Skip to content

Fix AI tools fallback and embed resolved skills releases - #7162

Closed
simonfaltum wants to merge 5 commits into
mainfrom
simonfaltum/aitools-build-skills-pin
Closed

simonfaltum wants to merge 5 commits into
mainfrom
simonfaltum/aitools-build-skills-pin

Conversation

@simonfaltum

@simonfaltum simonfaltum commented Oct 22, 2026 •

Copy link
Copy Markdown
Member

Why

CLI 1.22 resolves the skills version with an unauthenticated call to GitHub's releases API. When GitHub rate limits that call, aitools list and plugin installation fail, and the VS Code agent picker ends up empty (failing CI).

The CLI already has an embedded fallback for this, but those two commands didn't use it. The fallback was also a hand-maintained constant (0.2.30) that drifts behind the real releases.

Changes

Before: a rate-limited release lookup made aitools list and plugin installs fail, and the embedded fallback was a compiled constant.

Now: aitools list falls back to the embedded skills version, plugin installs succeed and record the plugin version as unknown, and release builds embed the skills release that is latest at build time.

  • .github/scripts/pin-cli-compat.py resolves "skills": "latest" with one authenticated gh api call and writes the result to a Go build overlay under .databricks/cli-compat/. GoReleaser and the build, snapshot and snapshot-release tasks build with that overlay. The checked-in cli-compat.json stays unchanged.
  • Resolved entries carry skills_resolved_from_latest: true, so the CLI still tracks latest at runtime and keeps its downgrade checks. The concrete version is only used as the fallback.
  • If the build-time lookup fails, "latest" stays in the manifest and the script warns. GoReleaser hides hook output, so release builds also write the warning to the GitHub Actions job summary.
  • Whenever the embedded manifest still contains "latest" (failed lookup or a plain go build), the fallback is the newest pinned range below it, as if the "latest" rows weren't there. This replaces AgentSkillsFallbackVersion.
  • aitools list uses GetSkillsRefWithFallback. Plugin installs don't need a skills version (the agent's marketplace picks it), so a failed lookup records the plugin version as unknown and keeps the existing state.Release.
  • Latest tracking starts at CLI 1.23.0. CLI 1.22.x stays pinned to 0.2.32 (Pin CLI 1.22 skills to v0.2.32 #7159), since those binaries don't have the fallback.
  • DATABRICKS_SKILLS_API_URL overrides the GitHub API base URL so tests can stub the release lookup.

Test plan

  • ./task checks, ./task lint-q, ./task fmt-q, ./task doctest
  • Unit tests for libs/clicompat, libs/aitools/..., cmd/aitools and internal/build (the first three also with the build overlay applied)
  • New acceptance test acceptance/cmd/aitools/list-release-rate-limited: a 403 from the release API still lists skills at 0.2.32, with the full agent list
  • New unit test: plugin install succeeds when the release lookup fails and records the plugin version as unknown
  • Script doctests: successful resolution, failed lookups (missing gh, non-zero exit, timeout, invalid tags) leave "latest" in place, relative overlay paths
  • Build through a symlinked checkout embeds the resolved manifest
  • Script with gh missing on Python 3.9: the build continues and the warning reaches stderr and the job summary

This PR was written by Codex.

This pull request and its description were written by Isaac.

@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Oct 22, 2026 •

Copy link
Copy Markdown
Collaborator

Integration test report

Commit: 065fd66

Run: 39783963165

Env ✅​pass 🙈​skip Time
✅​ aws linux-2core-8gb 281 16 4:17
✅​ aws-windows-latest-4core-16gb 283 14 3:15
✅​ azure linux-2core-8gb 280 16 4:41
✅​ azure-windows-latest-4core-16gb 282 14 3:08
✅​ gcp linux-2core-8gb 281 16 4:05
✅​ gcp-windows-latest-4core-16gb 283 14 3:08
Top 3 slowest tests (at least 2 minutes):
duration env testname
3:03 azure-windows-latest-4core-16gb TestAccept
3:02 aws-windows-latest-4core-16gb TestAccept
2:59 gcp-windows-latest-4core-16gb TestAccept

simonfaltum and others added 2 commits October 22, 2026 14:18
…uild-skills-pin

# Conflicts:
#	internal/build/cli-compat.json
#	libs/aitools/installer/installer_test.go
#	libs/clicompat/clicompat_test.go
- Write overlay paths relative to the build directory so symlinked
  checkouts still embed the resolved manifest.
- Keep "latest" when the build-time lookup fails, add a timeout to the
  gh call, and report the failure in the GitHub Actions job summary,
  since GoReleaser hides hook output.
- Resolve the embedded skills fallback from the newest pinned range
  below "latest" and remove AgentSkillsFallbackVersion.
- Record the plugin version as unknown when the latest release can't
  be resolved, instead of recording the fallback.
- Replace the libs-level command test with an acceptance test for
  aitools list and a cmd/aitools test for plugin installs. Both stub
  the release API through DATABRICKS_SKILLS_API_URL.

Co-authored-by: Isaac <no-reply@databricks.com>
@github-actions

Copy link
Copy Markdown
Contributor

Commit: 065fd664b | Workflow run

Regression test report

Selected 1 changed tests (limit=20, 0 not selected)

1 tests: regression=1 unreleased=0 coverage=0 failing=0

test branch main (66fce77) latest (v1.22.0)
TestAccept/cmd/aitools/list-release-rate-limited ✅ ❌ ❌
TestAccept/cmd/aitools/list-release-rate-limited: branch ✅ | main (66fce77) ❌ | latest (v1.22.0) ❌

main (66fce77):

acceptance_test.go:1204: Substituted DATABRICKS_SKILLS_API_URL "$DATABRICKS_HOST" -> "http://127.0.0.1:46259" ("[DATABRICKS_HOST]")
    acceptance_test.go:1204: Substituted DATABRICKS_SKILLS_BASE_URL "$DATABRICKS_HOST" -> "http://127.0.0.1:46259" ("[DATABRICKS_HOST]")
    acceptance_test.go:1307: 
        	Error Trace:	/home/runner/work/cli/cli/libs/testdiff/testdiff.go:22
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1307
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1077
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:637
        	Error:      	Not equal: 
        	            	expected: "\n=== list falls back to the embedded skills version when the release lookup is rate limited\n\n>>> [CLI] aitools list --output json\nWarn: Could not resolve latest skills release (HTTP 403 fetching latest skills release); falling back to embedded skills version 0.2.32\n{\n  \"release\": \"0.2.32\",\n  \"skills\": [\n    \"test-stable\"\n  ],\n  \"agents\": [\n    \"claude-code\",\n    \"cursor\",\n    \"codex\",\n    \"opencode\",\n    \"copilot\",\n    \"antigravity\",\n    \"pi\",\n    \"gemini\",\n    \"goose\",\n    \"kiro\"\n  ]\n}\n"
        	            	actual  : "\n=== list falls back to the embedded skills version when the release lookup is rate limited\n\n>>> [CLI] aitools list --output json\n{\n  \"release\": \"0.2.32\",\n  \"skills\": [\n    \"test-stable\"\n  ],\n  \"agents\": [\n    \"claude-code\",\n    \"cursor\",\n    \"codex\",\n    \"opencode\",\n    \"copilot\",\n    \"antigravity\",\n    \"pi\",\n    \"gemini\",\n    \"goose\",\n    \"kiro\"\n  ]\n}\n"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -4,3 +4,2 @@
        	            	 >>> [CLI] aitools list --output json
        	            	-Warn: Could not resolve latest skills release (HTTP 403 fetching latest skills release); falling back to embedded skills version 0.2.32
        	            	 {
        	Test:       	TestAccept/cmd/aitools/list-release-rate-limited
        	Messages:   	cmd/aitools/list-release-rate-limited/output.txt vs /tmp/TestAcceptcmdaitoolslist-release-rate-limited737567045/002/output.txt
FAIL

latest (v1.22.0):

acceptance_test.go:1204: Substituted DATABRICKS_SKILLS_API_URL "$DATABRICKS_HOST" -> "http://127.0.0.1:36733" ("[DATABRICKS_HOST]")
    acceptance_test.go:1204: Substituted DATABRICKS_SKILLS_BASE_URL "$DATABRICKS_HOST" -> "http://127.0.0.1:36733" ("[DATABRICKS_HOST]")
    acceptance_test.go:1307: 
        	Error Trace:	/home/runner/work/cli/cli/libs/testdiff/testdiff.go:22
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1307
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1077
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:637
        	Error:      	Not equal: 
        	            	expected: "\n=== list falls back to the embedded skills version when the release lookup is rate limited\n\n>>> [CLI] aitools list --output json\nWarn: Could not resolve latest skills release (HTTP 403 fetching latest skills release); falling back to embedded skills version 0.2.32\n{\n  \"release\": \"0.2.32\",\n  \"skills\": [\n    \"test-stable\"\n  ],\n  \"agents\": [\n    \"claude-code\",\n    \"cursor\",\n    \"codex\",\n    \"opencode\",\n    \"copilot\",\n    \"antigravity\",\n    \"pi\",\n    \"gemini\",\n    \"goose\",\n    \"kiro\"\n  ]\n}\n"
        	            	actual  : "\n=== list falls back to the embedded skills version when the release lookup is rate limited\n\n>>> [CLI] aitools list --output json\nError: resolve latest skills release: Get \"https://api.github.com/repos/databricks/databricks-agent-skills/releases/latest\": Bad Request\n\nExit code: 1\n"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -4,21 +4,5 @@
        	            	 >>> [CLI] aitools list --output json
        	            	-Warn: Could not resolve latest skills release (HTTP 403 fetching latest skills release); falling back to embedded skills version 0.2.32
        	            	-{
        	            	-  "release": "0.2.32",
        	            	-  "skills": [
        	            	-    "test-stable"
        	            	-  ],
        	            	-  "agents": [
        	            	-    "claude-code",
        	            	-    "cursor",
        	            	-    "codex",
        	            	-    "opencode",
        	            	-    "copilot",
        	            	-    "antigravity",
        	            	-    "pi",
        	            	-    "gemini",
        	            	-    "goose",
        	            	-    "kiro"
        	            	-  ]
        	            	-}
        	            	+Error: resolve latest skills release: Get "https://api.github.com/repos/databricks/databricks-agent-skills/releases/latest": Bad Request
        	            	 
        	            	+Exit code: 1
        	            	+
        	Test:       	TestAccept/cmd/aitools/list-release-rate-limited
        	Messages:   	cmd/aitools/list-release-rate-limited/output.txt vs /tmp/TestAcceptcmdaitoolslist-release-rate-limited1155822872/002/output.txt
FAIL

@simonfaltum
simonfaltum marked this pull request as ready for review October 22, 2026 13:27
@simonfaltum
simonfaltum requested review from a team and lennartkats-db as code owners October 22, 2026 13:27
@github-actions

github-actions Bot commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

This PR has not received an update in a while. If you want to keep this PR open, please leave a comment below or push a new commit and auto-close will be canceled.

@github-actions github-actions Bot added the Stale label Nov 6, 2026
@github-actions github-actions Bot closed this Nov 13, 2026
idelafuente-oc pushed a commit to idelafuente-oc/cli that referenced this pull request Nov 14, 2026
…atabricks#7474)

Stacked on databricks#7473.

Stacked on: aitools: refresh only the Databricks marketplace, and do not
fail the install on it

## Summary

Plugin installs no longer fail when the lookup of the latest skills
release on api.github.com fails (for example HTTP 403 when
rate-limited). The agent's own CLI fetches the plugin from its
marketplace, so the skills ref only labels the recorded plugin version.
On lookup failure the CLI now records the plugin version as unknown and
keeps the scope's existing `state.Release`, the same way a failed
marketplace refresh already records "version unknown".

```diff
 executePlan (cmd/aitools/install.go)
   if plugin items
     ref = GetSkillsRef(ctx)
-    lookup error -> install fails
+    lookup error -> debug log, ref = "" (plugin version unknown)
     install plugins

 RecordPluginInstalls (libs/aitools/installer/plugin.go)
-  state.Release = ref
+  if ref != "": state.Release = ref
```

`DATABRICKS_SKILLS_REF` and a pinned compat version still take
precedence. Skills installs are unchanged.

This is the same handling databricks#7162 uses for plugin installs
(same code lines, comments and test names), so whichever lands first,
the other rebases trivially. The cmd test stubs the lookup through
`DATABRICKS_CLI_GITHUB_API_URL`, which the release lookup uses on this
base.

Scope: install only. `aitools list` and `aitools update` still fail or
skip work on a failed lookup; they are left for a follow-up.

Changelog fragment: `.nextchanges/cli/aitools-plugin-ref-fallback.md`.

## Evidence

New test `TestExecutePlanPluginWithUnresolvedSkillsRelease`
(cmd/aitools): compat says "latest", `DATABRICKS_CLI_GITHUB_API_URL`
points at a stub that returns 403, and `installPluginForAgentFn` /
`recordPluginInstallsFn` are stubbed to capture the ref. It asserts the
command succeeds, the plugin is installed, the lookup ran once, and both
functions receive an empty ref.

- **Before** (the call site returning the lookup error, i.e. the old
behavior):
  ```
  --- FAIL: TestExecutePlanPluginWithUnresolvedSkillsRelease
Received unexpected error: resolve latest skills release: HTTP 403
fetching latest skills release
  ```
**After** (this head): `--- PASS:
TestExecutePlanPluginWithUnresolvedSkillsRelease`

New test `TestRecordPluginInstallsUnresolvedRefKeepsRelease`
(libs/aitools/installer): a plugin record written with an empty ref
keeps the `state.Release` recorded by an earlier install.

Also passing at this head: `go build ./...`, `task test-exp-aitools`,
`task lint-q`, `task fmt-q`, `task checks`, and the `cmd/aitools`
acceptance tests.

## Merge Danger

**Door:** two-way

A few lines at the plugin call site and in `RecordPluginInstalls`.
Reverting restores the old behavior.

**Blast Radius:** small

- Behavior change: when the lookup fails, `databricks aitools install`
with plugin targets exits 0 and shows the plugin version as unknown
instead of failing. `state.Release` is not overwritten.
- A mixed install (plugin agents plus skills agents) still runs the
skills group first, and that group can still fail on the lookup before
the plugin step.
- No extra network calls or waits: the lookup still runs once.
- Only unit-tested with a stubbed lookup. Not tested against a real
rate-limited GitHub API.

This pull request and its description were written by Isaac.

Co-authored-by: Isaac <no-reply@databricks.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants