Skip to content

[draft] Allow target overrides for complex-variable job tasks - #7347

Open
lennartkats-db wants to merge 2 commits into
mainfrom
factory/dabs/issue-2436-task-variable-overrides
Open

lennartkats-db wants to merge 2 commits into
mainfrom
factory/dabs/issue-2436-task-variable-overrides

Conversation

@lennartkats-db

@lennartkats-db lennartkats-db commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

Changes

Targets can override a task in a job whose tasks come from a direct complex variable reference. The other tasks remain in the job, and jobs sharing the variable keep their original task values. The task-list merge runs after variable assignment and before the existing task-key merge.

Why

In #2436, target selection tries to merge the unresolved ${var.job_tasks} string with a task sequence and fails with cannot merge string with sequence. This prevents a target from changing one task's cluster in a shared task list. Target values remain available to configuration rewrites before the deferred merge.

This change covers a direct ${var.name} task-list reference with a literal target task sequence. Transitive references and other resource fields are outside its scope.

Fixes #2436

Tests

  • go test ./bundle/config/... -count=1
  • go test ./acceptance -run TestAccept/bundle/variables/issue_2436 -tail -test.v (both local variants)
  • Current-head CI passed lint, Linux/Windows/macOS test variants, and integration tests. The regression report shows the acceptance fixture failing on main with the reported merge error and passing on this PR.

This PR was written by Codex.

@github-actions github-actions Bot added the DABs DABs related issues label Nov 4, 2026
@lennartkats-db lennartkats-db added the software-factory PR produced by the software factories task label Nov 4, 2026
@lennartkats-db
lennartkats-db marked this pull request as ready for review November 4, 2026 14:36
@lennartkats-db
lennartkats-db requested review from a team as code owners November 4, 2026 14:36
@lennartkats-db
lennartkats-db requested review from Sankalp-Mittal and rclarey and removed request for Sankalp-Mittal and rclarey November 4, 2026 14:36
@github-actions

github-actions Bot commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

Commit: 98256b344 | Workflow run

Regression test report

Selected 1 changed tests (limit=20, 0 not selected)

2 tests: regression=2 unreleased=0 coverage=0 failing=0
test branch main (eb1328d) latest (v1.24.0)
TestAccept/bundle/variables/issue_2436/DMS= ✅ ❌ ❌
TestAccept/bundle/variables/issue_2436/DMS=true ✅ ❌ ❌
TestAccept/bundle/variables/issue_2436/DMS=: branch ✅ | main (eb1328d) ❌ | latest (v1.24.0) ❌

main (eb1328d):

acceptance_test.go:1345: 
        	Error Trace:	/home/runner/work/cli/cli/libs/testdiff/testdiff.go:22
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1345
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1086
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:660
        	Error:      	Not equal: 
        	            	expected: "\n>>> [CLI] bundle validate -o json\nWarning: Ignoring Databricks CLI version constraint for development build. Required: >= 0.229.0, current: [CLI_VERSION]\n\n{\n  \"job_one\": [\n    {\n      \"task_key\": \"Task_One\",\n      \"job_cluster_key\": \"Medium\"\n    },\n    {\n      \"task_key\": \"Task_Two\",\n      \"job_cluster_key\": \"Small\"\n    }\n  ],\n  \"job_two\": [\n    {\n      \"task_key\": \"Task_One\",\n      \"job_cluster_key\": \"Small\"\n    },\n    {\n      \"task_key\": \"Task_Two\",\n      \"job_cluster_key\": \"Small\"\n    }\n  ]\n}\n"
        	            	actual  : "\n>>> [CLI] bundle validate -o json\nWarning: Ignoring Databricks CLI version constraint for development build. Required: >= 0.229.0, current: [CLI_VERSION]\n\nError: failed to perform target override for target=dev: failed to merge target=dev field=resources: cannot merge string with sequence\n\njq: error (at <stdin>:98): Cannot iterate over string (\"${var.job_...)\n\nExit code: 5\n"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -4,24 +4,7 @@
        	            	 
        	            	-{
        	            	-  "job_one": [
        	            	-    {
        	            	-      "task_key": "Task_One",
        	            	-      "job_cluster_key": "Medium"
        	            	-    },
        	            	-    {
        	            	-      "task_key": "Task_Two",
        	            	-      "job_cluster_key": "Small"
        	            	-    }
        	            	-  ],
        	            	-  "job_two": [
        	            	-    {
        	            	-      "task_key": "Task_One",
        	            	-      "job_cluster_key": "Small"
        	            	-    },
        	            	-    {
        	            	-      "task_key": "Task_Two",
        	            	-      "job_cluster_key": "Small"
        	            	-    }
        	            	-  ]
        	            	-}
        	            	+Error: failed to perform target override for target=dev: failed to merge target=dev field=resources: cannot merge string with sequence
        	            	 
        	            	+jq: error (at <stdin>:98): Cannot iterate over string ("${var.job_...)
        	            	+
        	            	+Exit code: 5
        	            	+
        	Test:       	TestAccept/bundle/variables/issue_2436/DMS=
        	Messages:   	bundle/variables/issue_2436/output.txt vs /tmp/TestAcceptbundlevariablesissue_2436DMS=1375562813/002/output.txt

latest (v1.24.0):

acceptance_test.go:1345: 
        	Error Trace:	/home/runner/work/cli/cli/libs/testdiff/testdiff.go:22
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1345
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1086
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:660
        	Error:      	Not equal: 
        	            	expected: "\n>>> [CLI] bundle validate -o json\nWarning: Ignoring Databricks CLI version constraint for development build. Required: >= 0.229.0, current: [CLI_VERSION]\n\n{\n  \"job_one\": [\n    {\n      \"task_key\": \"Task_One\",\n      \"job_cluster_key\": \"Medium\"\n    },\n    {\n      \"task_key\": \"Task_Two\",\n      \"job_cluster_key\": \"Small\"\n    }\n  ],\n  \"job_two\": [\n    {\n      \"task_key\": \"Task_One\",\n      \"job_cluster_key\": \"Small\"\n    },\n    {\n      \"task_key\": \"Task_Two\",\n      \"job_cluster_key\": \"Small\"\n    }\n  ]\n}\n"
        	            	actual  : "\n>>> [CLI] bundle validate -o json\nError: failed to perform target override for target=dev: failed to merge target=dev field=resources: cannot merge string with sequence\n\njq: error (at <stdin>:98): Cannot iterate over string (\"${var.job_...)\n\nExit code: 5\n"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -2,26 +2,7 @@
        	            	 >>> [CLI] bundle validate -o json
        	            	-Warning: Ignoring Databricks CLI version constraint for development build. Required: >= 0.229.0, current: [CLI_VERSION]
        	            	+Error: failed to perform target override for target=dev: failed to merge target=dev field=resources: cannot merge string with sequence
        	            	 
        	            	-{
        	            	-  "job_one": [
        	            	-    {
        	            	-      "task_key": "Task_One",
        	            	-      "job_cluster_key": "Medium"
        	            	-    },
        	            	-    {
        	            	-      "task_key": "Task_Two",
        	            	-      "job_cluster_key": "Small"
        	            	-    }
        	            	-  ],
        	            	-  "job_two": [
        	            	-    {
        	            	-      "task_key": "Task_One",
        	            	-      "job_cluster_key": "Small"
        	            	-    },
        	            	-    {
        	            	-      "task_key": "Task_Two",
        	            	-      "job_cluster_key": "Small"
        	            	-    }
        	            	-  ]
        	            	-}
        	            	+jq: error (at <stdin>:98): Cannot iterate over string ("${var.job_...)
        	            	 
        	            	+Exit code: 5
        	            	+
        	Test:       	TestAccept/bundle/variables/issue_2436/DMS=
        	Messages:   	bundle/variables/issue_2436/output.txt vs /tmp/TestAcceptbundlevariablesissue_2436DMS=4000496220/002/output.txt
TestAccept/bundle/variables/issue_2436/DMS=true: branch ✅ | main (eb1328d) ❌ | latest (v1.24.0) ❌

main (eb1328d):

acceptance_test.go:1345: 
        	Error Trace:	/home/runner/work/cli/cli/libs/testdiff/testdiff.go:22
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1345
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1086
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:660
        	Error:      	Not equal: 
        	            	expected: "\n>>> [CLI] bundle validate -o json\nWarning: Ignoring Databricks CLI version constraint for development build. Required: >= 0.229.0, current: [CLI_VERSION]\n\n{\n  \"job_one\": [\n    {\n      \"task_key\": \"Task_One\",\n      \"job_cluster_key\": \"Medium\"\n    },\n    {\n      \"task_key\": \"Task_Two\",\n      \"job_cluster_key\": \"Small\"\n    }\n  ],\n  \"job_two\": [\n    {\n      \"task_key\": \"Task_One\",\n      \"job_cluster_key\": \"Small\"\n    },\n    {\n      \"task_key\": \"Task_Two\",\n      \"job_cluster_key\": \"Small\"\n    }\n  ]\n}\n"
        	            	actual  : "\n>>> [CLI] bundle validate -o json\nWarning: Ignoring Databricks CLI version constraint for development build. Required: >= 0.229.0, current: [CLI_VERSION]\n\nError: failed to perform target override for target=dev: failed to merge target=dev field=resources: cannot merge string with sequence\n\njq: error (at <stdin>:98): Cannot iterate over string (\"${var.job_...)\n\nExit code: 5\n"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -4,24 +4,7 @@
        	            	 
        	            	-{
        	            	-  "job_one": [
        	            	-    {
        	            	-      "task_key": "Task_One",
        	            	-      "job_cluster_key": "Medium"
        	            	-    },
        	            	-    {
        	            	-      "task_key": "Task_Two",
        	            	-      "job_cluster_key": "Small"
        	            	-    }
        	            	-  ],
        	            	-  "job_two": [
        	            	-    {
        	            	-      "task_key": "Task_One",
        	            	-      "job_cluster_key": "Small"
        	            	-    },
        	            	-    {
        	            	-      "task_key": "Task_Two",
        	            	-      "job_cluster_key": "Small"
        	            	-    }
        	            	-  ]
        	            	-}
        	            	+Error: failed to perform target override for target=dev: failed to merge target=dev field=resources: cannot merge string with sequence
        	            	 
        	            	+jq: error (at <stdin>:98): Cannot iterate over string ("${var.job_...)
        	            	+
        	            	+Exit code: 5
        	            	+
        	Test:       	TestAccept/bundle/variables/issue_2436/DMS=true
        	Messages:   	bundle/variables/issue_2436/output.txt vs /tmp/TestAcceptbundlevariablesissue_2436DMS=true1397646324/002/output.txt
FAIL

latest (v1.24.0):

acceptance_test.go:1345: 
        	Error Trace:	/home/runner/work/cli/cli/libs/testdiff/testdiff.go:22
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1345
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1086
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:660
        	Error:      	Not equal: 
        	            	expected: "\n>>> [CLI] bundle validate -o json\nWarning: Ignoring Databricks CLI version constraint for development build. Required: >= 0.229.0, current: [CLI_VERSION]\n\n{\n  \"job_one\": [\n    {\n      \"task_key\": \"Task_One\",\n      \"job_cluster_key\": \"Medium\"\n    },\n    {\n      \"task_key\": \"Task_Two\",\n      \"job_cluster_key\": \"Small\"\n    }\n  ],\n  \"job_two\": [\n    {\n      \"task_key\": \"Task_One\",\n      \"job_cluster_key\": \"Small\"\n    },\n    {\n      \"task_key\": \"Task_Two\",\n      \"job_cluster_key\": \"Small\"\n    }\n  ]\n}\n"
        	            	actual  : "\n>>> [CLI] bundle validate -o json\nError: failed to perform target override for target=dev: failed to merge target=dev field=resources: cannot merge string with sequence\n\njq: error (at <stdin>:98): Cannot iterate over string (\"${var.job_...)\n\nExit code: 5\n"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -2,26 +2,7 @@
        	            	 >>> [CLI] bundle validate -o json
        	            	-Warning: Ignoring Databricks CLI version constraint for development build. Required: >= 0.229.0, current: [CLI_VERSION]
        	            	+Error: failed to perform target override for target=dev: failed to merge target=dev field=resources: cannot merge string with sequence
        	            	 
        	            	-{
        	            	-  "job_one": [
        	            	-    {
        	            	-      "task_key": "Task_One",
        	            	-      "job_cluster_key": "Medium"
        	            	-    },
        	            	-    {
        	            	-      "task_key": "Task_Two",
        	            	-      "job_cluster_key": "Small"
        	            	-    }
        	            	-  ],
        	            	-  "job_two": [
        	            	-    {
        	            	-      "task_key": "Task_One",
        	            	-      "job_cluster_key": "Small"
        	            	-    },
        	            	-    {
        	            	-      "task_key": "Task_Two",
        	            	-      "job_cluster_key": "Small"
        	            	-    }
        	            	-  ]
        	            	-}
        	            	+jq: error (at <stdin>:98): Cannot iterate over string ("${var.job_...)
        	            	 
        	            	+Exit code: 5
        	            	+
        	Test:       	TestAccept/bundle/variables/issue_2436/DMS=true
        	Messages:   	bundle/variables/issue_2436/output.txt vs /tmp/TestAcceptbundlevariablesissue_2436DMS=true1273844601/002/output.txt
FAIL

@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Nov 4, 2026 •

Copy link
Copy Markdown
Collaborator

✅ Integration test report

Commit: 98256b3

Run: 42040757537

Env 🔄​flaky ✅​pass 🙈​skip Time
✅​ aws linux-2core-8gb 271 16 5:15
✅​ aws-windows-latest-4core-16gb 273 14 4:32
🔄​ azure linux-2core-8gb 2 268 16 5:37
✅​ azure-windows-latest-4core-16gb 272 14 3:56
✅​ gcp linux-2core-8gb 271 16 5:14
✅​ gcp-windows-latest-4core-16gb 273 14 4:31
Test Name azure linux-2core-8gb
🔄​ TestFilerReadDir 🔄​f
🔄​ TestFilerReadDir/files 🔄​f
Top 3 slowest tests (at least 2 minutes):
duration env testname
3:08 aws-windows-latest-4core-16gb TestAccept
3:02 gcp-windows-latest-4core-16gb TestAccept
3:01 azure-windows-latest-4core-16gb TestAccept

@lennartkats-db lennartkats-db added the ai-auto PRs made by AI loops & factories, label Nov 13, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ai-auto PRs made by AI loops & factories, DABs DABs related issues software-factory PR produced by the software factories task

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Can not define tasks as a variable and then overwrite them

2 participants