Skip to content

air: refuse to upload an empty code_source snapshot - #7570

Merged
dimk0 merged 3 commits into
mainfrom
air-reject-empty-snapshot
Nov 12, 2026
Merged

dimk0 merged 3 commits into
mainfrom
air-reject-empty-snapshot

Conversation

@dimk0

@dimk0 dimk0 commented Nov 12, 2026

Copy link
Copy Markdown
Member

Changes

  • air run now refuses to upload a code_source snapshot that selects no files. The error names the cause: the git ignore rule covering root_path (with the git check-ignore -v command to inspect it), include_paths that match nothing, or an empty directory.
  • air run --dry-run now runs the snapshot file selection. It fails with the same errors as a real submission and otherwise reports the file count (snapshot_file_count in JSON output).

Why

air run selects snapshot files with git ls-files --cached --others --exclude-standard. When the directory holding the workload config is ignored by an enclosing repo (a scratch/ or tmp/ folder is common), the selection is empty. The CLI uploaded an empty tarball without warning, and the run failed on the node at extraction, after compute was provisioned, and was retried under max_retries. --dry-run did not catch it because packaging was not part of the dry run.

Behavior change: --dry-run with git.branch or git.commit now resolves the ref locally, so it needs a real repository. The air/run acceptance script sets one up for the existing with-code-source.yaml case.

Not in this PR: treating code-source extraction failures as non-retryable. That is a backend change.

Tests

  • Unit tests for the three empty-selection cases (git-ignored root, empty directory, include_paths matching nothing) and for the dry-run file count in text and JSON output.
  • The air/run acceptance test gains a case that reproduces the git-ignored root_path and asserts the error, and the existing code_source dry-run case now asserts the reported file count.
  • go test ./cmd/air and go test ./acceptance -run 'TestAccept/air/' pass.

This pull request and its description were written by Isaac.

dimk0 and others added 2 commits November 12, 2026 20:00
When the directory holding the workload config is ignored by an enclosing
repo, `git ls-files` selects nothing and `air run` uploaded an empty
tarball. The run then failed on the node at extraction, after compute was
provisioned, and was retried under max_retries.

Reject a snapshot that selects no files and name the cause: the git ignore
rule covering root_path, include_paths that match nothing, or an empty
directory. `--dry-run` now runs the same file selection and reports the
file count, so the problem is caught before submitting.

Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
@github-actions github-actions Bot added the AIR Databricks AI Runtime CLI label Nov 12, 2026
@dimk0
dimk0 marked this pull request as ready for review November 12, 2026 20:04
@dimk0
dimk0 requested review from a team as code owners November 12, 2026 20:04
@dimk0
dimk0 requested a review from anton-107 November 12, 2026 20:04
@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Commit: f719e353a | Workflow run

Regression test report

Selected 1 changed tests (limit=20, 0 not selected)

1 tests: regression=1 unreleased=0 coverage=0 failing=0
test branch main (b48c9fb) latest (v1.26.0)
TestAccept/air/run ✅ ❌ ❌
TestAccept/air/run: branch ✅ | main (b48c9fb) ❌ | latest (v1.26.0) ❌

main (b48c9fb):

acceptance_test.go:1374: Diff:
        --- air/run/output.txt
        +++ /tmp/TestAcceptairrun782444469/002/output.txt
        @@ -52,19 +52,13 @@
         === code_source config passes validation and reports the selected files
         
         >>> [CLI] air run -f with-code-source.yaml --dry-run
        -Dry run: code_source snapshot would include 1 file(s).
         Dry run: configuration for "smoke-test" is valid; not submitting.
         
         === git-ignored root_path is rejected instead of selecting no files
         
         >>> [CLI] air run -f ignored-root.yaml --dry-run
        -Error: code_source snapshot of [TEST_TMP_DIR]/.scratch/proj contains no files; refusing to upload an empty snapshot: root_path is ignored by git (.gitignore:1:.scratch/), and the snapshot only includes files git does not ignore. Move the project out of the ignored directory, un-ignore it, or change code_source.root_path. Inspect the rule with `git -C [TEST_TMP_DIR]/.scratch/proj check-ignore -v .`
        +Dry run: configuration for "smoke-test" is valid; not submitting.
         
        -Tip: use the --debug flag to see more details and a trace of this error:
        -  databricks --debug air run …
        -
        -Exit code: 1
        -
         === git.remote is rejected
         
         >>> [CLI] air run -f git-remote.yaml --dry-run
        
FAIL

latest (v1.26.0): same output as main (b48c9fb)

@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Collaborator

✅ Integration test report

Commit: f719e35

Run: 43306210443

Env 🔄​flaky 💚​RECOVERED ✅​pass 🙈​skip Time
💚​ aws linux-2core-8gb 1 270 16 13:01
🔄​ aws-windows-latest-4core-16gb 2 1 270 14 9:29
💚​ azure linux-2core-8gb 1 269 16 12:49
💚​ azure-windows-latest-4core-16gb 1 271 14 10:40
💚​ gcp linux-2core-8gb 1 270 16 14:17
🔄​ gcp-windows-latest-4core-16gb 1 1 271 14 12:53
Test Name aws linux-2core-8gb aws-windows-latest-4core-16gb azure linux-2core-8gb azure-windows-latest-4core-16gb gcp linux-2core-8gb gcp-windows-latest-4core-16gb
💚​ TestAccept 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
🔄​ TestSyncFullFileSync ✅​p ✅​p ✅​p ✅​p ✅​p 🔄​f
🔄​ TestSyncIncrementalFileOverwritesFolder ✅​p 🔄​f ✅​p ✅​p ✅​p ✅​p
🔄​ TestSyncIncrementalFileSync ✅​p 🔄​f ✅​p ✅​p ✅​p ✅​p
Top 26 slowest tests (at least 2 minutes):
duration env testname
5:26 gcp-windows-latest-4core-16gb TestImportDirWithOverwriteFlag
5:16 azure-windows-latest-4core-16gb TestFilerWorkspaceFilesExtensionsStat
4:11 azure linux-2core-8gb TestFilerWorkspaceFilesExtensionsRead
3:56 gcp linux-2core-8gb TestAccept
3:55 azure linux-2core-8gb TestAccept
3:54 aws linux-2core-8gb TestAccept
3:50 azure-windows-latest-4core-16gb TestAccept
3:48 gcp linux-2core-8gb TestFilerWorkspaceNotebook/sqlJupyterNotebook.ipynb
3:37 gcp-windows-latest-4core-16gb TestAccept
3:32 aws-windows-latest-4core-16gb TestAccept
3:23 aws linux-2core-8gb TestFilerWorkspaceFilesExtensionsDelete
2:48 gcp-windows-latest-4core-16gb TestFilerWorkspaceFilesExtensionsDelete
2:35 azure linux-2core-8gb TestExportDir
2:34 gcp linux-2core-8gb TestFilerWorkspaceFilesExtensionsRead
2:30 gcp linux-2core-8gb TestFilerWorkspaceFilesExtensionsReadDir
2:25 azure linux-2core-8gb TestFilerWorkspaceFilesExtensionsDelete
2:24 aws linux-2core-8gb TestFilerRecursiveDelete/workspace_files
2:21 gcp-windows-latest-4core-16gb TestFilerWorkspaceFilesExtensionsStat
2:16 azure linux-2core-8gb TestFilerRecursiveDelete/workspace_files
2:16 azure-windows-latest-4core-16gb TestImportDir
2:09 aws-windows-latest-4core-16gb TestFilerRecursiveDelete/workspace_files
2:08 gcp-windows-latest-4core-16gb TestFilerWorkspaceNotebook/rJupyterNb.ipynb
2:07 azure-windows-latest-4core-16gb TestWorkspaceFilesExtensions_ExportFormatIsPreserved/jupyter_r
2:05 gcp linux-2core-8gb TestFilerWorkspaceFilesExtensionsStat
2:03 gcp-windows-latest-4core-16gb TestFilerWorkspaceFilesExtensionsReadDir
2:03 azure-windows-latest-4core-16gb TestFilerWorkspaceFilesExtensionsReadDir

Comment thread cmd/air/snapshot_git.go Resolved
…path

Co-authored-by: Isaac <no-reply@databricks.com>
@dimk0
dimk0 added this pull request to the merge queue Nov 12, 2026
Merged via the queue into main with commit 14b822e Nov 12, 2026
28 checks passed
@dimk0
dimk0 deleted the air-reject-empty-snapshot branch November 12, 2026 22:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

AIR Databricks AI Runtime CLI

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants