Skip to content

acc: retry the spark-jar-task job task once when its cluster is lost - #7562

Merged
denik merged 1 commit into
mainfrom
denik/acc-spark-jar-task-retry
Nov 12, 2026
Merged

denik merged 1 commit into
mainfrom
denik/acc-spark-jar-task-retry

Conversation

@denik

@denik denik commented Nov 12, 2026

Copy link
Copy Markdown
Contributor

Changes

Set max_retries: 1 on the job task in acceptance/bundle/deploy/spark-jar-task, and raise its TimeoutCloud to 40m so a second job cluster has time to start.

Why

On AWS this test fails when its job cluster is lost, not because of a CLI problem. The AWS test instance pool is spot-only in a single zone, so a cluster can be spot-killed, refused for lack of capacity, or terminated while it installs the jar library. This test runs a full job cluster and is the slowest cluster test in the suite, so it hits these the most. One retry of the task on a new cluster covers a single lost cluster.

The golden output stays the same when the retry happens. bundle run prints output for every task attempt that has logs (bundle/run/output/job.go). An attempt that lost its cluster before the jar ran has no logs, so it is skipped. I checked this on AWS with a throwaway job whose task failed on purpose on the first attempt and passed on the second: the run stayed RUNNING and then ended TERMINATED SUCCESS, with no extra state lines.

Judgment calls:

  • I made the fix in the test, not the pool. Moving the pool to on-demand, spot-with-fallback, or auto-zone would fix this for every cluster test, but that is an infra change outside this repo.
  • One retry only. A capacity outage that lasts hours still fails the test, and those failures stay visible.

Tests

  • Local: go test ./acceptance -run TestAccept/bundle/deploy/spark-jar-task passes.
  • AWS: testme-aws -run TestAccept/bundle/deploy/spark-jar-task passes for both DMS= and DMS=true. The last commit names the test with aws: so the PR's cloud run includes it.

Dashboard

link failure
error/1691 INTERNAL_ERROR FAILED Task TestSparkJarTask failed with message: Library installation failed for library due to infra fault. Error messages: Failed due to cluster termination, cluster was in state: Terminated.
error/1687 Cluster '[ID]' was terminated. Reason: SPOT_INSTANCE_TERMINATION (CLIENT_ERROR). Parameters: databricks_error_message:Your VM is spot killed.
error/1688 Cluster [ID] was terminated during the run (cluster state message: driver is lost).
error/1579 Cluster '[ID]' was terminated. Reason: AWS_INSUFFICIENT_INSTANCE_CAPACITY_FAILURE (CLIENT_ERROR). (helps only when the capacity shortage is brief)

Tests: TestAccept/bundle/deploy/spark-jar-task/DMS=, TestAccept/bundle/deploy/spark-jar-task/DMS=true

This pull request and its description were written by Isaac.

The AWS test instance pool is spot-only in a single zone, so the job
cluster is sometimes lost (spot kill, no capacity, terminated during
library install). A lost cluster leaves no task logs, so the retried
run prints the same output as a clean one.

aws: TestAccept/bundle/deploy/spark-jar-task

Co-authored-by: Isaac <no-reply@databricks.com>
@denik denik added the ai-auto PRs made by AI loops & factories, label Nov 12, 2026
@github-actions github-actions Bot added DABs DABs related issues test-only-fix PRs that update failing of flaky tests without changing CLI code. labels Nov 12, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Commit: 0913e5507 | Workflow run

Regression test report

Selected 1 changed tests (limit=20, 0 not selected)

2 tests: regression=0 unreleased=0 coverage=2 failing=0
test branch main (2848f68) latest (v1.26.0)
TestAccept/bundle/deploy/spark-jar-task/DMS= ✅ ✅ ✅
TestAccept/bundle/deploy/spark-jar-task/DMS=true ✅ ✅ ✅

@eng-dev-ecosystem-bot

Copy link
Copy Markdown
Collaborator

✅ Integration test report

Commit: 0913e55

Run: 43236975522

Env 🔄​flaky 💚​RECOVERED ✅​pass 🙈​skip Time
💚​ aws linux-2core-8gb 1 273 15 14:31
💚​ aws-windows-latest-4core-16gb 1 275 13 9:56
💚​ azure linux-2core-8gb 1 272 15 15:14
💚​ azure-windows-latest-4core-16gb 1 274 13 12:10
💚​ gcp linux-2core-8gb 1 273 15 13:19
🔄​ gcp-windows-latest-4core-16gb 1 1 274 13 11:15
Test Name aws linux-2core-8gb aws-windows-latest-4core-16gb azure linux-2core-8gb azure-windows-latest-4core-16gb gcp linux-2core-8gb gcp-windows-latest-4core-16gb
💚​ TestAccept 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
🔄​ TestSyncIncrementalFileOverwritesFolder ✅​p ✅​p ✅​p ✅​p ✅​p 🔄​f
Top 27 slowest tests (at least 2 minutes):
duration env testname
9:40 azure linux-2core-8gb TestAccept/bundle/deploy/spark-jar-task/DMS=
8:36 azure-windows-latest-4core-16gb TestAccept/bundle/deploy/spark-jar-task/DMS=
6:55 gcp-windows-latest-4core-16gb TestAccept/bundle/deploy/spark-jar-task/DMS=true
6:09 aws linux-2core-8gb TestAccept/bundle/deploy/spark-jar-task/DMS=
5:37 azure-windows-latest-4core-16gb TestAccept/bundle/deploy/spark-jar-task/DMS=true
5:26 azure linux-2core-8gb TestAccept/bundle/deploy/spark-jar-task/DMS=true
4:40 aws linux-2core-8gb TestAccept/bundle/deploy/spark-jar-task/DMS=true
4:30 gcp-windows-latest-4core-16gb TestAccept/bundle/deploy/spark-jar-task/DMS=
4:19 aws-windows-latest-4core-16gb TestAccept/bundle/deploy/spark-jar-task/DMS=true
4:19 gcp linux-2core-8gb TestAccept/bundle/deploy/spark-jar-task/DMS=true
4:07 aws-windows-latest-4core-16gb TestAccept/bundle/deploy/spark-jar-task/DMS=
3:58 gcp linux-2core-8gb TestAccept/bundle/deploy/spark-jar-task/DMS=
3:51 aws-windows-latest-4core-16gb TestFilerWorkspaceFilesExtensionsDelete
3:47 aws-windows-latest-4core-16gb TestAccept
3:44 gcp linux-2core-8gb TestAccept
3:43 azure linux-2core-8gb TestAccept
3:32 aws linux-2core-8gb TestImportDirDoesNotOverwrite
3:31 gcp-windows-latest-4core-16gb TestAccept
3:31 azure-windows-latest-4core-16gb TestAccept
3:17 aws linux-2core-8gb TestAccept
2:47 azure-windows-latest-4core-16gb TestImportDirWithOverwriteFlag
2:28 aws linux-2core-8gb TestFilerWorkspaceFilesExtensionsReadDir
2:18 gcp-windows-latest-4core-16gb TestFilerWorkspaceFilesExtensionsStat
2:09 aws linux-2core-8gb TestFilerWorkspaceFilesExtensionsDelete
2:03 gcp-windows-latest-4core-16gb TestFilerWorkspaceFilesExtensionsDelete
2:02 azure-windows-latest-4core-16gb TestFilerWorkspaceFilesExtensionsReadDir
2:01 aws-windows-latest-4core-16gb TestFilerRecursiveDelete/workspace_files_extensions

@denik
denik marked this pull request as ready for review November 12, 2026 14:53
@denik
denik requested review from a team as code owners November 12, 2026 14:53
@denik
denik requested review from Sankalp-Mittal, andrewnester and rugpanov and removed request for Sankalp-Mittal November 12, 2026 14:53
@denik
denik added this pull request to the merge queue Nov 12, 2026
Merged via the queue into main with commit 60c0428 Nov 12, 2026
28 checks passed
@denik
denik deleted the denik/acc-spark-jar-task-retry branch November 12, 2026 15:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ai-auto PRs made by AI loops & factories, DABs DABs related issues test-only-fix PRs that update failing of flaky tests without changing CLI code.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants