Skip to content

Do not report single-node cluster settings supplied by the backend as drift - #7577

Open
andrewnester wants to merge 6 commits into
mainfrom
factory/7563-single-node-spark-conf
Open

andrewnester wants to merge 6 commits into
mainfrom
factory/7563-single-node-spark-conf

Conversation

@andrewnester

@andrewnester andrewnester commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Changes

Fixes #7563. bundle plan no longer reports the spark_conf and custom_tags the backend derives from is_single_node: true on a kind: CLASSIC_PREVIEW cluster as drift.

  • Declare those keys as backend_defaults for standalone clusters; explicitly configured keys still participate in drift detection.
  • Teach the fake Clusters API to expand is_single_node on create and edit.

Why

The backend adds spark.databricks.cluster.profile, spark.master and custom_tags.ResourceClass server-side, so a config that never declared them showed an update on every plan. The Jobs API stores cluster specs as authored, so job clusters are not affected.

Tests

New acceptance test bundle/resources/clusters/single_node (Cloud, CloudSlow), with the node-type dependent core count normalised.

This PR was written by Claude Code.
This pull request and its description were written by Isaac.

… drift

When a cluster spec sets is_single_node: true with kind: CLASSIC_PREVIEW, the
backend fills in spark_conf (spark.databricks.cluster.profile=singleNode,
spark.master=local[*, N]) and custom_tags.ResourceClass=SingleNode, and they
read back next to the flag. The direct engine planner compared that against a
config that never declared them and reported `spark_conf: {...} -> <unset>`
and the same for custom_tags on every `bundle plan`, for standalone clusters
and for job clusters alike. Unlike the cluster-policy case there is no policy
to gate on, so the existing ignore_remote_additions rule did not apply.

Declare those three keys as backend_defaults (pinned to their known values
where the value is fixed) for clusters and for every job cluster-spec
location. This only fires when the key is absent from both old and new
state, so an explicitly configured spark_conf or custom_tags key still
participates in drift detection. A when_set: is_single_node remote-addition
gate was rejected because it would hide every out-of-band addition on
single-node clusters, not just the three settings the flag generates.
Pipeline clusters have no is_single_node field, so nothing changes there.

The fake server now expands is_single_node the same way on cluster create
and edit and on job cluster specs, so the acceptance tests reproduce the
drift.

Fixes #7563

Co-authored-by: Isaac <no-reply@databricks.com>
@andrewnester andrewnester added the ai-auto PRs made by AI loops & factories, label Nov 12, 2026
@github-actions github-actions Bot added the DABs DABs related issues label Nov 12, 2026
@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Commit: a2ae76d2a | Workflow run

Regression test report

Selected 1 changed tests (limit=20, 0 not selected)

2 tests: regression=2 unreleased=0 coverage=0 failing=0
test branch main (eef3b54) latest (v1.26.0)
TestAccept/bundle/resources/clusters/single_node/DMS= ✅ ❌ ❌
TestAccept/bundle/resources/clusters/single_node/DMS=true ✅ ❌ ❌
TestAccept/bundle/resources/clusters/single_node/DMS=: branch ✅ | main (eef3b54) ❌ | latest (v1.26.0) ❌

main (eef3b54):

acceptance_test.go:1374: Diff:
        --- bundle/resources/clusters/single_node/output.txt
        +++ /tmp/TestAcceptbundleresourcesclusterssingle_nodeDMS=832955146/002/output.txt
        @@ -9,42 +9,40 @@
         === Settings the backend derives from is_single_node are not drift
         
         >>> [CLI] bundle plan
        -Plan: 0 to add, 0 to change, 0 to delete, 2 unchanged
        +update clusters.explicit
        +update clusters.implicit
         
        +Plan: 0 to add, 2 to change, 0 to delete, 0 unchanged
        +
         === ...classified backend_default
         
         >>> [CLI] bundle plan -o json
         {
           "resources.clusters.explicit": {
             "custom_tags": {
        -      "action": "skip",
        -      "reason": "backend_default",
        +      "action": "update",
               "remote": {
                 "ResourceClass": "SingleNode"
               }
             },
             "spark_conf['spark.databricks.cluster.profile']": {
        -      "action": "skip",
        -      "reason": "backend_default",
        +      "action": "update",
               "remote": "singleNode"
             },
             "spark_conf['spark.master']": {
        -      "action": "skip",
        -      "reason": "backend_default",
        +      "action": "update",
               "remote": "local[*, [CORES]]"
             }
           },
           "resources.clusters.implicit": {
             "custom_tags": {
        -      "action": "skip",
        -      "reason": "backend_default",
        +      "action": "update",
               "remote": {
                 "ResourceClass": "SingleNode"
               }
             },
             "spark_conf": {
        -      "action": "skip",
        -      "reason": "backend_default",
        +      "action": "update",
               "remote": {
                 "spark.databricks.cluster.profile": "singleNode",
                 "spark.master": "local[*, [CORES]]"
        @@ -57,19 +30,18 @@
         
         >>> [CLI] bundle plan
         update clusters.explicit
        +update clusters.implicit
         
        -Plan: 0 to add, 1 to change, 0 to delete, 1 unchanged
        +Plan: 0 to add, 2 to change, 0 to delete, 0 unchanged
         
         >>> [CLI] bundle plan -o json
         {
           "spark_conf['spark.databricks.cluster.profile']": {
        -    "action": "skip",
        -    "reason": "backend_default",
        +    "action": "update",
             "remote": "singleNode"
           },
           "spark_conf['spark.master']": {
        -    "action": "skip",
        -    "reason": "backend_default",
        +    "action": "update",
             "remote": "local[*, [CORES]]"
           },
           "spark_conf['spark.sql.shuffle.partitions']": {
        
    acceptance_test.go:1152: 
        LOG.edit: 
        LOG.edit: >>> /tmp/TestAccept1568488918/001/databricks clusters edit --json @edit.json
        LOG.edit: {
        LOG.edit:   "autotermination_minutes": 10,
        LOG.edit:   "aws_attributes": {
        LOG.edit:     "availability": "SPOT_WITH_FALLBACK",
        LOG.edit:     "zone_id": "us-east-1c"
        LOG.edit:   },
        LOG.edit:   "cluster_id": "7a0c95ed-be23-1943-7a0c-95edbe231a5b",
        LOG.edit:   "cluster_name": "test-cluster-explicit-3i7pjepcrjdfjb6u7be32buo5a",
        LOG.edit:   "custom_tags": {
        LOG.edit:     "ResourceClass": "SingleNode"
        LOG.edit:   },
        LOG.edit:   "data_security_mode": "SINGLE_USER",
        LOG.edit:   "driver_node_type_id": "i3.xlarge",
        LOG.edit:   "enable_elastic_disk": false,
        LOG.edit:   "is_single_node": true,
        LOG.edit:   "kind": "CLASSIC_PREVIEW",
        LOG.edit:   "node_type_id": "i3.xlarge",
        LOG.edit:   "spark_conf": {
        LOG.edit:     "spark.databricks.cluster.profile": "singleNode",
        LOG.edit:     "spark.master": "local[*, 4]",
        LOG.edit:     "spark.sql.shuffle.partitions": "16"
        LOG.edit:   },
        LOG.edit:   "spark_version": "13.3.x-snapshot-scala2.12",
        LOG.edit:   "state": "RUNNING"
        LOG.edit: }

latest (v1.26.0):

acceptance_test.go:1374: Diff:
        --- bundle/resources/clusters/single_node/output.txt
        +++ /tmp/TestAcceptbundleresourcesclusterssingle_nodeDMS=3064363560/002/output.txt
        @@ -9,42 +9,40 @@
         === Settings the backend derives from is_single_node are not drift
         
         >>> [CLI] bundle plan
        -Plan: 0 to add, 0 to change, 0 to delete, 2 unchanged
        +update clusters.explicit
        +update clusters.implicit
         
        +Plan: 0 to add, 2 to change, 0 to delete, 0 unchanged
        +
         === ...classified backend_default
         
         >>> [CLI] bundle plan -o json
         {
           "resources.clusters.explicit": {
             "custom_tags": {
        -      "action": "skip",
        -      "reason": "backend_default",
        +      "action": "update",
               "remote": {
                 "ResourceClass": "SingleNode"
               }
             },
             "spark_conf['spark.databricks.cluster.profile']": {
        -      "action": "skip",
        -      "reason": "backend_default",
        +      "action": "update",
               "remote": "singleNode"
             },
             "spark_conf['spark.master']": {
        -      "action": "skip",
        -      "reason": "backend_default",
        +      "action": "update",
               "remote": "local[*, [CORES]]"
             }
           },
           "resources.clusters.implicit": {
             "custom_tags": {
        -      "action": "skip",
        -      "reason": "backend_default",
        +      "action": "update",
               "remote": {
                 "ResourceClass": "SingleNode"
               }
             },
             "spark_conf": {
        -      "action": "skip",
        -      "reason": "backend_default",
        +      "action": "update",
               "remote": {
                 "spark.databricks.cluster.profile": "singleNode",
                 "spark.master": "local[*, [CORES]]"
        @@ -57,19 +30,18 @@
         
         >>> [CLI] bundle plan
         update clusters.explicit
        +update clusters.implicit
         
        -Plan: 0 to add, 1 to change, 0 to delete, 1 unchanged
        +Plan: 0 to add, 2 to change, 0 to delete, 0 unchanged
         
         >>> [CLI] bundle plan -o json
         {
           "spark_conf['spark.databricks.cluster.profile']": {
        -    "action": "skip",
        -    "reason": "backend_default",
        +    "action": "update",
             "remote": "singleNode"
           },
           "spark_conf['spark.master']": {
        -    "action": "skip",
        -    "reason": "backend_default",
        +    "action": "update",
             "remote": "local[*, [CORES]]"
           },
           "spark_conf['spark.sql.shuffle.partitions']": {
        
    acceptance_test.go:1152: 
        LOG.edit: 
        LOG.edit: >>> /home/runner/work/cli/cli/acceptance/build/linux_amd64/1.26.0/databricks clusters edit --json @edit.json
        LOG.edit: {
        LOG.edit:   "autotermination_minutes": 10,
        LOG.edit:   "aws_attributes": {
        LOG.edit:     "availability": "SPOT_WITH_FALLBACK",
        LOG.edit:     "zone_id": "us-east-1c"
        LOG.edit:   },
        LOG.edit:   "cluster_id": "7a0c95ee-9f1b-ee07-7a0c-95ee9f1beebc",
        LOG.edit:   "cluster_name": "test-cluster-explicit-6ku2wluxfzer7ofyajvfedljn4",
        LOG.edit:   "custom_tags": {
        LOG.edit:     "ResourceClass": "SingleNode"
        LOG.edit:   },
        LOG.edit:   "data_security_mode": "SINGLE_USER",
        LOG.edit:   "driver_node_type_id": "i3.xlarge",
        LOG.edit:   "enable_elastic_disk": false,
        LOG.edit:   "is_single_node": true,
        LOG.edit:   "kind": "CLASSIC_PREVIEW",
        LOG.edit:   "node_type_id": "i3.xlarge",
        LOG.edit:   "spark_conf": {
        LOG.edit:     "spark.databricks.cluster.profile": "singleNode",
        LOG.edit:     "spark.master": "local[*, 4]",
        LOG.edit:     "spark.sql.shuffle.partitions": "16"
        LOG.edit:   },
        LOG.edit:   "spark_version": "13.3.x-snapshot-scala2.12",
        LOG.edit:   "state": "RUNNING"
        LOG.edit: }
TestAccept/bundle/resources/clusters/single_node/DMS=true: branch ✅ | main (eef3b54) ❌ | latest (v1.26.0) ❌

main (eef3b54):

acceptance_test.go:1374: Diff:
        --- bundle/resources/clusters/single_node/output.txt
        +++ /tmp/TestAcceptbundleresourcesclusterssingle_nodeDMS=true305560987/002/output.txt
        @@ -9,42 +9,40 @@
         === Settings the backend derives from is_single_node are not drift
         
         >>> [CLI] bundle plan
        -Plan: 0 to add, 0 to change, 0 to delete, 2 unchanged
        +update clusters.explicit
        +update clusters.implicit
         
        +Plan: 0 to add, 2 to change, 0 to delete, 0 unchanged
        +
         === ...classified backend_default
         
         >>> [CLI] bundle plan -o json
         {
           "resources.clusters.explicit": {
             "custom_tags": {
        -      "action": "skip",
        -      "reason": "backend_default",
        +      "action": "update",
               "remote": {
                 "ResourceClass": "SingleNode"
               }
             },
             "spark_conf['spark.databricks.cluster.profile']": {
        -      "action": "skip",
        -      "reason": "backend_default",
        +      "action": "update",
               "remote": "singleNode"
             },
             "spark_conf['spark.master']": {
        -      "action": "skip",
        -      "reason": "backend_default",
        +      "action": "update",
               "remote": "local[*, [CORES]]"
             }
           },
           "resources.clusters.implicit": {
             "custom_tags": {
        -      "action": "skip",
        -      "reason": "backend_default",
        +      "action": "update",
               "remote": {
                 "ResourceClass": "SingleNode"
               }
             },
             "spark_conf": {
        -      "action": "skip",
        -      "reason": "backend_default",
        +      "action": "update",
               "remote": {
                 "spark.databricks.cluster.profile": "singleNode",
                 "spark.master": "local[*, [CORES]]"
        @@ -57,19 +30,18 @@
         
         >>> [CLI] bundle plan
         update clusters.explicit
        +update clusters.implicit
         
        -Plan: 0 to add, 1 to change, 0 to delete, 1 unchanged
        +Plan: 0 to add, 2 to change, 0 to delete, 0 unchanged
         
         >>> [CLI] bundle plan -o json
         {
           "spark_conf['spark.databricks.cluster.profile']": {
        -    "action": "skip",
        -    "reason": "backend_default",
        +    "action": "update",
             "remote": "singleNode"
           },
           "spark_conf['spark.master']": {
        -    "action": "skip",
        -    "reason": "backend_default",
        +    "action": "update",
             "remote": "local[*, [CORES]]"
           },
           "spark_conf['spark.sql.shuffle.partitions']": {
        
    acceptance_test.go:1152: 
        LOG.edit: 
        LOG.edit: >>> /tmp/TestAccept1568488918/001/databricks clusters edit --json @edit.json
        LOG.edit: {
        LOG.edit:   "autotermination_minutes": 10,
        LOG.edit:   "aws_attributes": {
        LOG.edit:     "availability": "SPOT_WITH_FALLBACK",
        LOG.edit:     "zone_id": "us-east-1c"
        LOG.edit:   },
        LOG.edit:   "cluster_id": "7a0c95ed-be75-c1e4-7a0c-95edbe75c28e",
        LOG.edit:   "cluster_name": "test-cluster-explicit-byxxjxy7ova6zmxfm57qo6yhvi",
        LOG.edit:   "custom_tags": {
        LOG.edit:     "ResourceClass": "SingleNode"
        LOG.edit:   },
        LOG.edit:   "data_security_mode": "SINGLE_USER",
        LOG.edit:   "driver_node_type_id": "i3.xlarge",
        LOG.edit:   "enable_elastic_disk": false,
        LOG.edit:   "is_single_node": true,
        LOG.edit:   "kind": "CLASSIC_PREVIEW",
        LOG.edit:   "node_type_id": "i3.xlarge",
        LOG.edit:   "spark_conf": {
        LOG.edit:     "spark.databricks.cluster.profile": "singleNode",
        LOG.edit:     "spark.master": "local[*, 4]",
        LOG.edit:     "spark.sql.shuffle.partitions": "16"
        LOG.edit:   },
        LOG.edit:   "spark_version": "13.3.x-snapshot-scala2.12",
        LOG.edit:   "state": "RUNNING"
        LOG.edit: }
FAIL

latest (v1.26.0):

acceptance_test.go:1374: Diff:
        --- bundle/resources/clusters/single_node/output.txt
        +++ /tmp/TestAcceptbundleresourcesclusterssingle_nodeDMS=true889926218/002/output.txt
        @@ -9,42 +9,40 @@
         === Settings the backend derives from is_single_node are not drift
         
         >>> [CLI] bundle plan
        -Plan: 0 to add, 0 to change, 0 to delete, 2 unchanged
        +update clusters.explicit
        +update clusters.implicit
         
        +Plan: 0 to add, 2 to change, 0 to delete, 0 unchanged
        +
         === ...classified backend_default
         
         >>> [CLI] bundle plan -o json
         {
           "resources.clusters.explicit": {
             "custom_tags": {
        -      "action": "skip",
        -      "reason": "backend_default",
        +      "action": "update",
               "remote": {
                 "ResourceClass": "SingleNode"
               }
             },
             "spark_conf['spark.databricks.cluster.profile']": {
        -      "action": "skip",
        -      "reason": "backend_default",
        +      "action": "update",
               "remote": "singleNode"
             },
             "spark_conf['spark.master']": {
        -      "action": "skip",
        -      "reason": "backend_default",
        +      "action": "update",
               "remote": "local[*, [CORES]]"
             }
           },
           "resources.clusters.implicit": {
             "custom_tags": {
        -      "action": "skip",
        -      "reason": "backend_default",
        +      "action": "update",
               "remote": {
                 "ResourceClass": "SingleNode"
               }
             },
             "spark_conf": {
        -      "action": "skip",
        -      "reason": "backend_default",
        +      "action": "update",
               "remote": {
                 "spark.databricks.cluster.profile": "singleNode",
                 "spark.master": "local[*, [CORES]]"
        @@ -57,19 +30,18 @@
         
         >>> [CLI] bundle plan
         update clusters.explicit
        +update clusters.implicit
         
        -Plan: 0 to add, 1 to change, 0 to delete, 1 unchanged
        +Plan: 0 to add, 2 to change, 0 to delete, 0 unchanged
         
         >>> [CLI] bundle plan -o json
         {
           "spark_conf['spark.databricks.cluster.profile']": {
        -    "action": "skip",
        -    "reason": "backend_default",
        +    "action": "update",
             "remote": "singleNode"
           },
           "spark_conf['spark.master']": {
        -    "action": "skip",
        -    "reason": "backend_default",
        +    "action": "update",
             "remote": "local[*, [CORES]]"
           },
           "spark_conf['spark.sql.shuffle.partitions']": {
        
    acceptance_test.go:1152: 
        LOG.edit: 
        LOG.edit: >>> /home/runner/work/cli/cli/acceptance/build/linux_amd64/1.26.0/databricks clusters edit --json @edit.json
        LOG.edit: {
        LOG.edit:   "autotermination_minutes": 10,
        LOG.edit:   "aws_attributes": {
        LOG.edit:     "availability": "SPOT_WITH_FALLBACK",
        LOG.edit:     "zone_id": "us-east-1c"
        LOG.edit:   },
        LOG.edit:   "cluster_id": "7a0c95ee-a5f3-4435-7a0c-95eea5f34589",
        LOG.edit:   "cluster_name": "test-cluster-explicit-xxhdbucuevhq5cmki56za5zjty",
        LOG.edit:   "custom_tags": {
        LOG.edit:     "ResourceClass": "SingleNode"
        LOG.edit:   },
        LOG.edit:   "data_security_mode": "SINGLE_USER",
        LOG.edit:   "driver_node_type_id": "i3.xlarge",
        LOG.edit:   "enable_elastic_disk": false,
        LOG.edit:   "is_single_node": true,
        LOG.edit:   "kind": "CLASSIC_PREVIEW",
        LOG.edit:   "node_type_id": "i3.xlarge",
        LOG.edit:   "spark_conf": {
        LOG.edit:     "spark.databricks.cluster.profile": "singleNode",
        LOG.edit:     "spark.master": "local[*, 4]",
        LOG.edit:     "spark.sql.shuffle.partitions": "16"
        LOG.edit:   },
        LOG.edit:   "spark_version": "13.3.x-snapshot-scala2.12",
        LOG.edit:   "state": "RUNNING"
        LOG.edit: }
FAIL

@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Collaborator

✅ Integration test report

Commit: a2ae76d

Run: 43415973769

Env 🔄​flaky 💚​RECOVERED ✅​pass 🙈​skip Time
💚​ aws linux-2core-8gb 1 270 16 7:08
💚​ aws-windows-latest-4core-16gb 1 272 14 5:15
💚​ azure linux-2core-8gb 1 269 16 8:30
💚​ azure-windows-latest-4core-16gb 1 271 14 5:00
💚​ gcp linux-2core-8gb 1 270 16 8:06
🔄​ gcp-windows-latest-4core-16gb 1 1 271 14 6:01
Test Name aws linux-2core-8gb aws-windows-latest-4core-16gb azure linux-2core-8gb azure-windows-latest-4core-16gb gcp linux-2core-8gb gcp-windows-latest-4core-16gb
💚​ TestAccept 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
🔄​ TestSyncFullFileSync ✅​p ✅​p ✅​p ✅​p ✅​p 🔄​f
Top 6 slowest tests (at least 2 minutes):
duration env testname
4:10 aws linux-2core-8gb TestAccept
3:55 azure-windows-latest-4core-16gb TestAccept
3:47 azure linux-2core-8gb TestAccept
3:47 gcp linux-2core-8gb TestAccept
3:46 gcp-windows-latest-4core-16gb TestAccept
3:42 aws-windows-latest-4core-16gb TestAccept

@andrewnester
andrewnester marked this pull request as ready for review November 13, 2026 10:30
@andrewnester
andrewnester requested review from a team as code owners November 13, 2026 10:30
andrewnester and others added 2 commits November 13, 2026 11:17
…er specs

Against a real workspace the Jobs API stores new_cluster specs as authored,
so the backend_defaults rules for job clusters never matched and the
jobs/single_node cloud run failed with an empty plan. Drop the rules, the
fake expansion on job cluster specs, and the test that depended on them.

Co-authored-by: Isaac <no-reply@databricks.com>
Normalise the node-type dependent core count in spark.master and stop
re-sending the backend-derived spark_conf keys on clusters edit.

Co-authored-by: Isaac <no-reply@databricks.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ai-auto PRs made by AI loops & factories, DABs DABs related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bundle plan proposes removing backend-generated spark_conf for is_single_node clusters

2 participants