Repository navigation
Do not report single-node cluster settings supplied by the backend as drift - #7577
Open
andrewnester wants to merge 6 commits into
Open
andrewnester wants to merge 6 commits into
andrewnester wants to merge 6 commits into
Conversation
… drift
When a cluster spec sets is_single_node: true with kind: CLASSIC_PREVIEW, the
backend fills in spark_conf (spark.databricks.cluster.profile=singleNode,
spark.master=local[*, N]) and custom_tags.ResourceClass=SingleNode, and they
read back next to the flag. The direct engine planner compared that against a
config that never declared them and reported `spark_conf: {...} -> <unset>`
and the same for custom_tags on every `bundle plan`, for standalone clusters
and for job clusters alike. Unlike the cluster-policy case there is no policy
to gate on, so the existing ignore_remote_additions rule did not apply.
Declare those three keys as backend_defaults (pinned to their known values
where the value is fixed) for clusters and for every job cluster-spec
location. This only fires when the key is absent from both old and new
state, so an explicitly configured spark_conf or custom_tags key still
participates in drift detection. A when_set: is_single_node remote-addition
gate was rejected because it would hide every out-of-band addition on
single-node clusters, not just the three settings the flag generates.
Pipeline clusters have no is_single_node field, so nothing changes there.
The fake server now expands is_single_node the same way on cluster create
and edit and on job cluster specs, so the acceptance tests reproduce the
drift.
Fixes #7563
Co-authored-by: Isaac <no-reply@databricks.com>
Contributor
|
Commit: Regression test reportSelected 1 changed tests (limit=20, 0 not selected) 2 tests: regression=2 unreleased=0 coverage=0 failing=0
TestAccept/bundle/resources/clusters/single_node/DMS=: branch ✅ | main (eef3b54) ❌ | latest (v1.26.0) ❌main (eef3b54): latest (v1.26.0): TestAccept/bundle/resources/clusters/single_node/DMS=true: branch ✅ | main (eef3b54) ❌ | latest (v1.26.0) ❌main (eef3b54): latest (v1.26.0): |
Collaborator
✅ Integration test reportCommit: a2ae76d
Top 6 slowest tests (at least 2 minutes):
|
andrewnester
marked this pull request as ready for review
November 13, 2026 10:30
…er specs Against a real workspace the Jobs API stores new_cluster specs as authored, so the backend_defaults rules for job clusters never matched and the jobs/single_node cloud run failed with an empty plan. Drop the rules, the fake expansion on job cluster specs, and the test that depended on them. Co-authored-by: Isaac <no-reply@databricks.com>
Normalise the node-type dependent core count in spark.master and stop re-sending the backend-derived spark_conf keys on clusters edit. Co-authored-by: Isaac <no-reply@databricks.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
Fixes #7563.
bundle planno longer reports thespark_confandcustom_tagsthe backend derives fromis_single_node: trueon akind: CLASSIC_PREVIEWcluster as drift.backend_defaultsfor standalone clusters; explicitly configured keys still participate in drift detection.is_single_nodeon create and edit.Why
The backend adds
spark.databricks.cluster.profile,spark.masterandcustom_tags.ResourceClassserver-side, so a config that never declared them showed an update on every plan. The Jobs API stores cluster specs as authored, so job clusters are not affected.Tests
New acceptance test
bundle/resources/clusters/single_node(Cloud, CloudSlow), with the node-type dependent core count normalised.This PR was written by Claude Code.
This pull request and its description were written by Isaac.