Repository navigation
acc: retry the spark-jar-task job task once when its cluster is lost - #7562
Merged
Merged
Conversation
The AWS test instance pool is spot-only in a single zone, so the job cluster is sometimes lost (spot kill, no capacity, terminated during library install). A lost cluster leaves no task logs, so the retried run prints the same output as a clean one. aws: TestAccept/bundle/deploy/spark-jar-task Co-authored-by: Isaac <no-reply@databricks.com>
Contributor
|
Commit: Regression test reportSelected 1 changed tests (limit=20, 0 not selected) 2 tests: regression=0 unreleased=0 coverage=2 failing=0
|
Collaborator
✅ Integration test reportCommit: 0913e55
Top 27 slowest tests (at least 2 minutes):
|
denik
marked this pull request as ready for review
November 12, 2026 14:53
denik
requested review from
Sankalp-Mittal,
andrewnester and
rugpanov
and removed request for
Sankalp-Mittal
November 12, 2026 14:53
andrewnester
approved these changes
Nov 12, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
Set
max_retries: 1on the job task inacceptance/bundle/deploy/spark-jar-task, and raise itsTimeoutCloudto 40m so a second job cluster has time to start.Why
On AWS this test fails when its job cluster is lost, not because of a CLI problem. The AWS test instance pool is spot-only in a single zone, so a cluster can be spot-killed, refused for lack of capacity, or terminated while it installs the jar library. This test runs a full job cluster and is the slowest cluster test in the suite, so it hits these the most. One retry of the task on a new cluster covers a single lost cluster.
The golden output stays the same when the retry happens.
bundle runprints output for every task attempt that has logs (bundle/run/output/job.go). An attempt that lost its cluster before the jar ran has no logs, so it is skipped. I checked this on AWS with a throwaway job whose task failed on purpose on the first attempt and passed on the second: the run stayedRUNNINGand then endedTERMINATED SUCCESS, with no extra state lines.Judgment calls:
Tests
go test ./acceptance -run TestAccept/bundle/deploy/spark-jar-taskpasses.testme-aws -run TestAccept/bundle/deploy/spark-jar-taskpasses for bothDMS=andDMS=true. The last commit names the test withaws:so the PR's cloud run includes it.Dashboard
INTERNAL_ERROR FAILED Task TestSparkJarTask failed with message: Library installation failed for library due to infra fault. Error messages: Failed due to cluster termination, cluster was in state: Terminated.Cluster '[ID]' was terminated. Reason: SPOT_INSTANCE_TERMINATION (CLIENT_ERROR). Parameters: databricks_error_message:Your VM is spot killed.Cluster [ID] was terminated during the run (cluster state message: driver is lost).Cluster '[ID]' was terminated. Reason: AWS_INSUFFICIENT_INSTANCE_CAPACITY_FAILURE (CLIENT_ERROR).(helps only when the capacity shortage is brief)Tests: TestAccept/bundle/deploy/spark-jar-task/DMS=, TestAccept/bundle/deploy/spark-jar-task/DMS=true
This pull request and its description were written by Isaac.