Skip to content

air: suggest a working --output-dir when convert-to-dabs code is outside the bundle - #7644

Draft
dimk0 wants to merge 5 commits into
mainfrom
cnxt-2650-convert-to-dabs-outside-bundle
Draft

dimk0 wants to merge 5 commits into
mainfrom
cnxt-2650-convert-to-dabs-outside-bundle

Conversation

@dimk0

@dimk0 dimk0 commented Nov 15, 2026 •

Copy link
Copy Markdown
Member

Changes

  • air convert-to-dabs: when code_source.root_path resolves outside the bundle directory, the error names the resolved code directory and, for working-tree snapshots, the --output-dir that converts the workload: the code directory's parent, quoted for the user's shell (cmd.exe or PowerShell on Windows, a POSIX shell elsewhere). It no longer suggests setting sync.paths in a databricks.yml it has not written.
  • When that parent already holds a bundle file (databricks.yml, databricks.yaml, bundle.yml or bundle.yaml), the error says so instead of suggesting it. Sibling workloads that share a code directory would otherwise each be sent there and replace one another's bundle.
  • A parent path that no quoting keeps literal, such as one with a control character, gets generic wording instead of a printed path.
  • quoteArg moves from cmd/aitools to libs/shellquote as Arg, unchanged apart from the name, so both commands quote printed paths the same way.
  • Git-pinned snapshots still get only "move the code inside the bundle directory", matching the rule for the bundle-root case in air: explain root_path and default retries in convert-to-dabs #7579.

Why

The getting-started Ray recipes in the AI Runtime cookbook snapshot their parent directory (root_path: ..) to include a shared bootstrap. Running convert-to-dabs workload.yaml from a recipe directory failed with advice to set sync.paths in a databricks.yml the command had not written; the conversion works with --output-dir ../.., which no message mentioned. Four recipes share that parent, so the hint must not steer a second conversion into the first one's bundle.

Replaces #7640, opened from a fork where CI could not get its OIDC token.

Tests

  • TestConvertToDabsCodeOutsideBundleSuggestsOutputDir: the error names the resolved code directory, --output-dir <parent> and the fallback, not sync.paths; converting with that directory succeeds with the expected artifact path and include.
  • TestConvertToDabsCodeOutsideBundleSkipsParentWithBundle: for each bundle file name, the error says the parent holds a bundle and does not suggest it.
  • TestConvertToDabsCodeOutsideBundleQuotesOutputDir: a parent path with a space is quoted for the platform's shell.
  • TestConvertToDabsGitPinOutsideBundleDoesNotSuggestOutputDir: git-pinned snapshots get only the move-the-code remedy. TestArg covers the moved helper.
  • Removing the bundle-file check or the quoting makes the corresponding test fail.
  • Ran the built CLI on a copy of the cookbook's Ray recipes: Ray Core suggests the parent and converts there, and Ray Data then reports the existing bundle.
  • go test ./cmd/air/... ./libs/shellquote/..., go test ./acceptance -run TestAccept/air, and ./task test-exp-aitools pass, and golangci-lint run reports no issues on the changed packages.
  • Not covered here: re-running a workload whose own bundle is in the parent is told to use another directory instead of --force, and --force still replaces a bundle with a different bundle.name. Both belong to a follow-up that compares bundle names.

This pull request and its description were written by Isaac.

dimk0 and others added 4 commits November 14, 2026 23:49
…ide the bundle

When code_source.root_path resolved outside the bundle directory, for example
the getting-started Ray recipes that snapshot their parent directory with
root_path: .., convert-to-dabs suggested setting sync.paths in a databricks.yml
that it had not written. The error now names the resolved code directory and,
for working-tree snapshots, the --output-dir value that converts the workload.

Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Sibling workloads that snapshot the same parent directory were all sent to it,
so converting a second one there replaced the first bundle once the user
passed --force. The hint now says when that directory already holds a
databricks.yml instead of offering it, quotes the suggested path for the
shell, and states the actual limit: the bundle convert-to-dabs writes packages
only code inside its directory. The tests assert the full remedy text.

Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Collaborator

✅ Integration test report

Commit: 025d2ab

Run: 43689513008

Env 💚​RECOVERED ✅​pass 🙈​skip Time
💚​ aws linux-2core-8gb 1 270 15 3:39
💚​ aws-windows-latest-4core-16gb 1 272 13 3:03
💚​ azure linux-2core-8gb 1 269 15 3:38
💚​ azure-windows-latest-4core-16gb 1 271 13 3:19
💚​ gcp linux-2core-8gb 1 270 15 3:24
💚​ gcp-windows-latest-4core-16gb 1 272 13 2:54
Test Name aws linux-2core-8gb aws-windows-latest-4core-16gb azure linux-2core-8gb azure-windows-latest-4core-16gb gcp linux-2core-8gb gcp-windows-latest-4core-16gb
💚​ TestAccept 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
Top 3 slowest tests (at least 2 minutes):
duration env testname
3:15 azure-windows-latest-4core-16gb TestAccept
2:59 aws-windows-latest-4core-16gb TestAccept
2:50 gcp-windows-latest-4core-16gb TestAccept

…ry bundle file name

- Move quoteArg from cmd/aitools to libs/shellquote as Arg, unchanged apart
  from the name, and use it for the suggested --output-dir. Bash quoting
  broke a pasted cmd.exe command on Windows; a path that no quoting keeps
  literal, such as one with a control character, gets generic wording.
- Treat a parent holding any bundle file DABs accepts (databricks.yml,
  databricks.yaml, bundle.yml, bundle.yaml) as already holding a bundle.
- Test a parent whose path has a space, and each bundle file name.

Co-authored-by: Isaac <no-reply@databricks.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

AIR Databricks AI Runtime CLI

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants