Skip to content

Add the testing property of pipelines to bundles - #7300

Draft
tharunbhuk-db wants to merge 1 commit into
tharun/pipelines-test-b1-editable-folderfrom
tharun/pipelines-test-pr4-bundles
Draft

tharunbhuk-db wants to merge 1 commit into
tharun/pipelines-test-b1-editable-folderfrom
tharun/pipelines-test-pr4-bundles

Conversation

@tharunbhuk-db

@tharunbhuk-db tharunbhuk-db commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

🥞 Stacked PR


  • Adds resources.pipelines.<key>.testing (sources, environment.dependencies). The CLI has its own types because the Go SDK does not have the field yet. The schema and Python models are generated.
  • Without the variable, bundle validate rejects testing. Bundles without testing do not change.

Tests: TestPipelineTestingEnabled; acceptance bundle/resources/pipelines/testing.

This pull request and its description were written by Isaac.

@github-actions

github-actions Bot commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

Commit: 86997dc92 | Workflow run

Regression test report

Selected 3 changed tests (limit=20, 0 not selected)

6 tests: regression=6 unreleased=0 coverage=0 failing=0
test branch main (cec3f63) latest (v1.26.0)
TestAccept/bundle/environments/dependencies/DMS= ✅ ❌ ❌
TestAccept/bundle/environments/dependencies/DMS=true ✅ ❌ ❌
TestAccept/bundle/refschema/DMS= ✅ ❌ ❌
TestAccept/bundle/refschema/DMS=true ✅ ❌ ❌
TestAccept/bundle/resources/pipelines/testing/DMS= ✅ ❌ ❌
TestAccept/bundle/resources/pipelines/testing/DMS=true ✅ ❌ ❌
TestAccept/bundle/environments/dependencies/DMS=: branch ✅ | main (cec3f63) ❌ | latest (v1.26.0) ❌

main (cec3f63):

acceptance_test.go:1345: Diff:
        --- bundle/environments/dependencies/output.txt
        +++ /tmp/TestAcceptbundleenvironmentsdependenciesDMS=549411206/002/output.txt
        @@ -1,150 +1,13 @@
         
         >>> [CLI] bundle validate
        +Error: unable to determine if [TEST_TMP_DIR]/project is not a notebook: read [TEST_TMP_DIR]/project: is a directory
        +
         Name: dependencies
         Target: default
         Workspace:
           User: [USERNAME]
           Path: /Workspace/Users/[USERNAME]/.bundle/dependencies/default
         
        -Validation OK!
        +Found 1 error
         
        ->>> [CLI] bundle deploy
        -Uploading dist/test.whl...
        -Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dependencies/default/files...
        -Created jobs.test_job
        -Created pipelines.test_pipeline
        -Files: 6 uploaded, 0 deleted
        -Resources: 2 created, 0 changed, 0 deleted, 0 unchanged
        -
        ->>> jq -s .[] | select(.path=="/api/2.2/jobs/create") | .body.environments [OUTPUT_DIR]/out.requests.txt
        -[
        -  {
        -    "environment_key": "test_env",
        -    "spec": {
        -      "client": "1",
        -      "dependencies": [
        -        "-r /Workspace/Users/[USERNAME]/.bundle/dependencies/default/files/requirements.txt",
        -        "-r \"/Workspace/Users/[USERNAME]/.bundle/dependencies/default/files/Path With Spaces/requirements.txt\"",
        -        "-r '/Workspace/Users/[USERNAME]/.bundle/dependencies/default/files/Path With Spaces/requirements.txt'",
        -        "-e /Workspace/Users/[USERNAME]/.bundle/dependencies/default/files/file.py",
        -        "-e /Workspace/Users/[USERNAME]/.bundle/dependencies/default/files/project",
        -        "-e /Workspace/Users/[USERNAME]/.bundle/dependencies/default/files/project[test]",
        -        "-i http://myindexurl.com",
        -        "--index-url http://myindexurl.com",
        -        "test_package",
        -        "test_package==2.0.1",
        -        "test_package>=2.0.1",
        -        "test_package==2.0.*",
        -        "/Workspace/Users/[USERNAME]/.bundle/dependencies/default/artifacts/.internal/test.whl",
        -        "/Workspace/Users/test@databricks.com/test-package.whl",
        -        "beautifulsoup4>=1.0.0,~=1.2.0,<2.0.0",
        -        "beautifulsoup4[security, tests] ~= 4.12.3",
        -        "requests[security] @ https://github.com/psf/requests/archive/refs/heads/main.zip",
        -        "--find-links=/Workspace/Users/test"
        -      ]
        -    }
        -  },
        -  {
        -    "environment_key": "test_env_2",
        -    "spec": {
        -      "client": "1",
        -      "dependencies": [
        -        "-r /Workspace/Users/[USERNAME]/.bundle/dependencies/default/files/requirements.txt"
        -      ]
        -    }
        -  }
        -]
        -
        ->>> jq -s .[] | select(.path=="/api/2.0/pipelines") [OUTPUT_DIR]/out.requests.txt
        -{
        -  "method": "POST",
        -  "path": "/api/2.0/pipelines",
        -  "body": {
        -    "channel": "CURRENT",
        -    "deployment": {
        -      "kind": "BUNDLE",
        -      "metadata_file_path": "/Workspace/Users/[USERNAME]/.bundle/dependencies/default/state/metadata.json"
        -    },
        -    "edition": "ADVANCED",
        -    "environment": {
        -      "dependencies": [
        -        "-r /Workspace/Users/[USERNAME]/.bundle/dependencies/default/files/requirements.txt",
        -        "-e /Workspace/Users/[USERNAME]/.bundle/dependencies/default/files/file.py",
        -        "-e /Workspace/Users/[USERNAME]/.bundle/dependencies/default/files/project",
        -        "-e /Workspace/Users/[USERNAME]/.bundle/dependencies/default/files/project[test]",
        -        "-i http://myindexurl.com",
        -        "--index-url http://myindexurl.com",
        -        "/Workspace/Users/[USERNAME]/.bundle/dependencies/default/artifacts/.internal/test.whl",
        -        "/Workspace/Users/test@databricks.com/test-package.whl",
        -        "test_package",
        -        "test_package==2.0.1",
        -        "test_package>=2.0.1",
        -        "test_package==2.0.*",
        -        "beautifulsoup4>=1.0.0,~=1.2.0,<2.0.0",
        -        "requests[security] @ https://github.com/psf/requests/archive/refs/heads/main.zip",
        -        "--find-links=/Workspace/Users/test"
        -      ]
        -    },
        -    "name": "Test Pipeline"
        -  }
        -}
        -
        ->>> [CLI] bundle validate -o json
        -[
        -  {
        -    "environment_key": "test_env",
        -    "spec": {
        -      "client": "1",
        -      "dependencies": [
        -        "-r /Workspace/Users/[USERNAME]/.bundle/dependencies/default/files/requirements.txt",
        -        "-r \"/Workspace/Users/[USERNAME]/.bundle/dependencies/default/files/Path With Spaces/requirements.txt\"",
        -        "-r '/Workspace/Users/[USERNAME]/.bundle/dependencies/default/files/Path With Spaces/requirements.txt'",
        -        "-e /Workspace/Users/[USERNAME]/.bundle/dependencies/default/files/file.py",
        -        "-e /Workspace/Users/[USERNAME]/.bundle/dependencies/default/files/project",
        -        "-e /Workspace/Users/[USERNAME]/.bundle/dependencies/default/files/project[test]",
        -        "-i http://myindexurl.com",
        -        "--index-url http://myindexurl.com",
        -        "test_package",
        -        "test_package==2.0.1",
        -        "test_package>=2.0.1",
        -        "test_package==2.0.*",
        -        "./dist/*.whl",
        -        "/Workspace/Users/test@databricks.com/test-package.whl",
        -        "beautifulsoup4>=1.0.0,~=1.2.0,<2.0.0",
        -        "beautifulsoup4[security, tests] ~= 4.12.3",
        -        "requests[security] @ https://github.com/psf/requests/archive/refs/heads/main.zip",
        -        "--find-links=/Workspace/Users/test"
        -      ]
        -    }
        -  },
        -  {
        -    "environment_key": "test_env_2",
        -    "spec": {
        -      "client": "1",
        -      "dependencies": [
        -        "-r /Workspace/Users/[USERNAME]/.bundle/dependencies/default/files/requirements.txt"
        -      ]
        -    }
        -  }
        -]
        -
        ->>> [CLI] bundle validate -o json
        -{
        -  "dependencies": [
        -    "-r /Workspace/Users/[USERNAME]/.bundle/dependencies/default/files/requirements.txt",
        -    "-e /Workspace/Users/[USERNAME]/.bundle/dependencies/default/files/file.py",
        -    "-e /Workspace/Users/[USERNAME]/.bundle/dependencies/default/files/project",
        -    "-e /Workspace/Users/[USERNAME]/.bundle/dependencies/default/files/project[test]",
        -    "-i http://myindexurl.com",
        -    "--index-url http://myindexurl.com",
        -    "./dist/*.whl",
        -    "/Workspace/Users/test@databricks.com/test-package.whl",
        -    "test_package",
        -    "test_package==2.0.1",
        -    "test_package>=2.0.1",
        -    "test_package==2.0.*",
        -    "beautifulsoup4>=1.0.0,~=1.2.0,<2.0.0",
        -    "requests[security] @ https://github.com/psf/requests/archive/refs/heads/main.zip",
        -    "--find-links=/Workspace/Users/test"
        -  ]
        -}
        +Exit code: 1

latest (v1.26.0): same output as main (cec3f63)

TestAccept/bundle/environments/dependencies/DMS=true: branch ✅ | main (cec3f63) ❌ | latest (v1.26.0) ❌

main (cec3f63):

acceptance_test.go:1345: Diff:
        --- bundle/environments/dependencies/output.txt
        +++ /tmp/TestAcceptbundleenvironmentsdependenciesDMS=true2661451169/002/output.txt
        @@ -1,150 +1,13 @@
         
         >>> [CLI] bundle validate
        +Error: unable to determine if [TEST_TMP_DIR]/project is not a notebook: read [TEST_TMP_DIR]/project: is a directory
        +
         Name: dependencies
         Target: default
         Workspace:
           User: [USERNAME]
           Path: /Workspace/Users/[USERNAME]/.bundle/dependencies/default
         
        -Validation OK!
        +Found 1 error
         
        ->>> [CLI] bundle deploy
        -Uploading dist/test.whl...
        -Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dependencies/default/files...
        -Created jobs.test_job
        -Created pipelines.test_pipeline
        -Files: 6 uploaded, 0 deleted
        -Resources: 2 created, 0 changed, 0 deleted, 0 unchanged
        -
        ->>> jq -s .[] | select(.path=="/api/2.2/jobs/create") | .body.environments [OUTPUT_DIR]/out.requests.txt
        -[
        -  {
        -    "environment_key": "test_env",
        -    "spec": {
        -      "client": "1",
        -      "dependencies": [
        -        "-r /Workspace/Users/[USERNAME]/.bundle/dependencies/default/files/requirements.txt",
        -        "-r \"/Workspace/Users/[USERNAME]/.bundle/dependencies/default/files/Path With Spaces/requirements.txt\"",
        -        "-r '/Workspace/Users/[USERNAME]/.bundle/dependencies/default/files/Path With Spaces/requirements.txt'",
        -        "-e /Workspace/Users/[USERNAME]/.bundle/dependencies/default/files/file.py",
        -        "-e /Workspace/Users/[USERNAME]/.bundle/dependencies/default/files/project",
        -        "-e /Workspace/Users/[USERNAME]/.bundle/dependencies/default/files/project[test]",
        -        "-i http://myindexurl.com",
        -        "--index-url http://myindexurl.com",
        -        "test_package",
        -        "test_package==2.0.1",
        -        "test_package>=2.0.1",
        -        "test_package==2.0.*",
        -        "/Workspace/Users/[USERNAME]/.bundle/dependencies/default/artifacts/.internal/test.whl",
        -        "/Workspace/Users/test@databricks.com/test-package.whl",
        -        "beautifulsoup4>=1.0.0,~=1.2.0,<2.0.0",
        -        "beautifulsoup4[security, tests] ~= 4.12.3",
        -        "requests[security] @ https://github.com/psf/requests/archive/refs/heads/main.zip",
        -        "--find-links=/Workspace/Users/test"
        -      ]
        -    }
        -  },
        -  {
        -    "environment_key": "test_env_2",
        -    "spec": {
        -      "client": "1",
        -      "dependencies": [
        -        "-r /Workspace/Users/[USERNAME]/.bundle/dependencies/default/files/requirements.txt"
        -      ]
        -    }
        -  }
        -]
        -
        ->>> jq -s .[] | select(.path=="/api/2.0/pipelines") [OUTPUT_DIR]/out.requests.txt
        -{
        -  "method": "POST",
        -  "path": "/api/2.0/pipelines",
        -  "body": {
        -    "channel": "CURRENT",
        -    "deployment": {
        -      "kind": "BUNDLE",
        -      "metadata_file_path": "/Workspace/Users/[USERNAME]/.bundle/dependencies/default/state/metadata.json"
        -    },
        -    "edition": "ADVANCED",
        -    "environment": {
        -      "dependencies": [
        -        "-r /Workspace/Users/[USERNAME]/.bundle/dependencies/default/files/requirements.txt",
        -        "-e /Workspace/Users/[USERNAME]/.bundle/dependencies/default/files/file.py",
        -        "-e /Workspace/Users/[USERNAME]/.bundle/dependencies/default/files/project",
        -        "-e /Workspace/Users/[USERNAME]/.bundle/dependencies/default/files/project[test]",
        -        "-i http://myindexurl.com",
        -        "--index-url http://myindexurl.com",
        -        "/Workspace/Users/[USERNAME]/.bundle/dependencies/default/artifacts/.internal/test.whl",
        -        "/Workspace/Users/test@databricks.com/test-package.whl",
        -        "test_package",
        -        "test_package==2.0.1",
        -        "test_package>=2.0.1",
        -        "test_package==2.0.*",
        -        "beautifulsoup4>=1.0.0,~=1.2.0,<2.0.0",
        -        "requests[security] @ https://github.com/psf/requests/archive/refs/heads/main.zip",
        -        "--find-links=/Workspace/Users/test"
        -      ]
        -    },
        -    "name": "Test Pipeline"
        -  }
        -}
        -
        ->>> [CLI] bundle validate -o json
        -[
        -  {
        -    "environment_key": "test_env",
        -    "spec": {
        -      "client": "1",
        -      "dependencies": [
        -        "-r /Workspace/Users/[USERNAME]/.bundle/dependencies/default/files/requirements.txt",
        -        "-r \"/Workspace/Users/[USERNAME]/.bundle/dependencies/default/files/Path With Spaces/requirements.txt\"",
        -        "-r '/Workspace/Users/[USERNAME]/.bundle/dependencies/default/files/Path With Spaces/requirements.txt'",
        -        "-e /Workspace/Users/[USERNAME]/.bundle/dependencies/default/files/file.py",
        -        "-e /Workspace/Users/[USERNAME]/.bundle/dependencies/default/files/project",
        -        "-e /Workspace/Users/[USERNAME]/.bundle/dependencies/default/files/project[test]",
        -        "-i http://myindexurl.com",
        -        "--index-url http://myindexurl.com",
        -        "test_package",
        -        "test_package==2.0.1",
        -        "test_package>=2.0.1",
        -        "test_package==2.0.*",
        -        "./dist/*.whl",
        -        "/Workspace/Users/test@databricks.com/test-package.whl",
        -        "beautifulsoup4>=1.0.0,~=1.2.0,<2.0.0",
        -        "beautifulsoup4[security, tests] ~= 4.12.3",
        -        "requests[security] @ https://github.com/psf/requests/archive/refs/heads/main.zip",
        -        "--find-links=/Workspace/Users/test"
        -      ]
        -    }
        -  },
        -  {
        -    "environment_key": "test_env_2",
        -    "spec": {
        -      "client": "1",
        -      "dependencies": [
        -        "-r /Workspace/Users/[USERNAME]/.bundle/dependencies/default/files/requirements.txt"
        -      ]
        -    }
        -  }
        -]
        -
        ->>> [CLI] bundle validate -o json
        -{
        -  "dependencies": [
        -    "-r /Workspace/Users/[USERNAME]/.bundle/dependencies/default/files/requirements.txt",
        -    "-e /Workspace/Users/[USERNAME]/.bundle/dependencies/default/files/file.py",
        -    "-e /Workspace/Users/[USERNAME]/.bundle/dependencies/default/files/project",
        -    "-e /Workspace/Users/[USERNAME]/.bundle/dependencies/default/files/project[test]",
        -    "-i http://myindexurl.com",
        -    "--index-url http://myindexurl.com",
        -    "./dist/*.whl",
        -    "/Workspace/Users/test@databricks.com/test-package.whl",
        -    "test_package",
        -    "test_package==2.0.1",
        -    "test_package>=2.0.1",
        -    "test_package==2.0.*",
        -    "beautifulsoup4>=1.0.0,~=1.2.0,<2.0.0",
        -    "requests[security] @ https://github.com/psf/requests/archive/refs/heads/main.zip",
        -    "--find-links=/Workspace/Users/test"
        -  ]
        -}
        +Exit code: 1

latest (v1.26.0): same output as main (cec3f63)

TestAccept/bundle/refschema/DMS=: branch ✅ | main (cec3f63) ❌ | latest (v1.26.0) ❌

main (cec3f63):

acceptance_test.go:1345: Diff:
        --- bundle/refschema/output/pipelines.txt
        +++ /tmp/TestAcceptbundlerefschemaDMS=4190759063/001/output/pipelines.txt
        @@ -867,14 +867,6 @@
         resources.pipelines.*.tags	map[string]string	ALL
         resources.pipelines.*.tags.*	string	ALL
         resources.pipelines.*.target	string	ALL
        -resources.pipelines.*.testing	*resources.PipelineTesting	INPUT
        -resources.pipelines.*.testing.environment	*resources.PipelineTestEnvironment	INPUT
        -resources.pipelines.*.testing.environment.dependencies	[]string	INPUT
        -resources.pipelines.*.testing.environment.dependencies[*]	string	INPUT
        -resources.pipelines.*.testing.sources	[]resources.PipelineTestSource	INPUT
        -resources.pipelines.*.testing.sources[*]	resources.PipelineTestSource	INPUT
        -resources.pipelines.*.testing.sources[*].glob	*pipelines.PathPattern	INPUT
        -resources.pipelines.*.testing.sources[*].glob.include	string	INPUT
         resources.pipelines.*.trigger	*pipelines.PipelineTrigger	ALL
         resources.pipelines.*.trigger.cron	*pipelines.CronTrigger	ALL
         resources.pipelines.*.trigger.cron.quartz_cron_schedule	string	ALL

latest (v1.26.0):

acceptance_test.go:1345: Diff:
        --- bundle/refschema/output/pipelines.txt
        +++ /tmp/TestAcceptbundlerefschemaDMS=4163172788/001/output/pipelines.txt
        @@ -867,14 +867,6 @@
         resources.pipelines.*.tags	map[string]string	ALL
         resources.pipelines.*.tags.*	string	ALL
         resources.pipelines.*.target	string	ALL
        -resources.pipelines.*.testing	*resources.PipelineTesting	INPUT
        -resources.pipelines.*.testing.environment	*resources.PipelineTestEnvironment	INPUT
        -resources.pipelines.*.testing.environment.dependencies	[]string	INPUT
        -resources.pipelines.*.testing.environment.dependencies[*]	string	INPUT
        -resources.pipelines.*.testing.sources	[]resources.PipelineTestSource	INPUT
        -resources.pipelines.*.testing.sources[*]	resources.PipelineTestSource	INPUT
        -resources.pipelines.*.testing.sources[*].glob	*pipelines.PathPattern	INPUT
        -resources.pipelines.*.testing.sources[*].glob.include	string	INPUT
         resources.pipelines.*.trigger	*pipelines.PipelineTrigger	ALL
         resources.pipelines.*.trigger.cron	*pipelines.CronTrigger	ALL
         resources.pipelines.*.trigger.cron.quartz_cron_schedule	string	ALL
        
    acceptance_test.go:1345: Diff:
        --- bundle/refschema/output/catalogs.txt
        +++ /tmp/TestAcceptbundlerefschemaDMS=4163172788/001/output/catalogs.txt
        @@ -12,7 +12,7 @@
         resources.catalogs.*.enable_predictive_optimization	catalog.EnablePredictiveOptimization	REMOTE
         resources.catalogs.*.full_name	string	REMOTE
         resources.catalogs.*.id	string	INPUT
        -resources.catalogs.*.isolation_mode	catalog.CatalogIsolationMode	ALL
        +resources.catalogs.*.isolation_mode	catalog.CatalogIsolationMode	REMOTE
         resources.catalogs.*.lifecycle	resources.Lifecycle	INPUT
         resources.catalogs.*.lifecycle.depends_on	[]resources.DependsOn	INPUT
         resources.catalogs.*.lifecycle.depends_on[*]	resources.DependsOn	INPUT
TestAccept/bundle/refschema/DMS=true: branch ✅ | main (cec3f63) ❌ | latest (v1.26.0) ❌

main (cec3f63):

acceptance_test.go:1345: Diff:
        --- bundle/refschema/output/pipelines.txt
        +++ /tmp/TestAcceptbundlerefschemaDMS=true1220140542/001/output/pipelines.txt
        @@ -867,14 +867,6 @@
         resources.pipelines.*.tags	map[string]string	ALL
         resources.pipelines.*.tags.*	string	ALL
         resources.pipelines.*.target	string	ALL
        -resources.pipelines.*.testing	*resources.PipelineTesting	INPUT
        -resources.pipelines.*.testing.environment	*resources.PipelineTestEnvironment	INPUT
        -resources.pipelines.*.testing.environment.dependencies	[]string	INPUT
        -resources.pipelines.*.testing.environment.dependencies[*]	string	INPUT
        -resources.pipelines.*.testing.sources	[]resources.PipelineTestSource	INPUT
        -resources.pipelines.*.testing.sources[*]	resources.PipelineTestSource	INPUT
        -resources.pipelines.*.testing.sources[*].glob	*pipelines.PathPattern	INPUT
        -resources.pipelines.*.testing.sources[*].glob.include	string	INPUT
         resources.pipelines.*.trigger	*pipelines.PipelineTrigger	ALL
         resources.pipelines.*.trigger.cron	*pipelines.CronTrigger	ALL
         resources.pipelines.*.trigger.cron.quartz_cron_schedule	string	ALL
        
FAIL

latest (v1.26.0):

acceptance_test.go:1345: Diff:
        --- bundle/refschema/output/catalogs.txt
        +++ /tmp/TestAcceptbundlerefschemaDMS=true1379698965/001/output/catalogs.txt
        @@ -12,7 +12,7 @@
         resources.catalogs.*.enable_predictive_optimization	catalog.EnablePredictiveOptimization	REMOTE
         resources.catalogs.*.full_name	string	REMOTE
         resources.catalogs.*.id	string	INPUT
        -resources.catalogs.*.isolation_mode	catalog.CatalogIsolationMode	ALL
        +resources.catalogs.*.isolation_mode	catalog.CatalogIsolationMode	REMOTE
         resources.catalogs.*.lifecycle	resources.Lifecycle	INPUT
         resources.catalogs.*.lifecycle.depends_on	[]resources.DependsOn	INPUT
         resources.catalogs.*.lifecycle.depends_on[*]	resources.DependsOn	INPUT
        
    acceptance_test.go:1345: Diff:
        --- bundle/refschema/output/pipelines.txt
        +++ /tmp/TestAcceptbundlerefschemaDMS=true1379698965/001/output/pipelines.txt
        @@ -867,14 +867,6 @@
         resources.pipelines.*.tags	map[string]string	ALL
         resources.pipelines.*.tags.*	string	ALL
         resources.pipelines.*.target	string	ALL
        -resources.pipelines.*.testing	*resources.PipelineTesting	INPUT
        -resources.pipelines.*.testing.environment	*resources.PipelineTestEnvironment	INPUT
        -resources.pipelines.*.testing.environment.dependencies	[]string	INPUT
        -resources.pipelines.*.testing.environment.dependencies[*]	string	INPUT
        -resources.pipelines.*.testing.sources	[]resources.PipelineTestSource	INPUT
        -resources.pipelines.*.testing.sources[*]	resources.PipelineTestSource	INPUT
        -resources.pipelines.*.testing.sources[*].glob	*pipelines.PathPattern	INPUT
        -resources.pipelines.*.testing.sources[*].glob.include	string	INPUT
         resources.pipelines.*.trigger	*pipelines.PipelineTrigger	ALL
         resources.pipelines.*.trigger.cron	*pipelines.CronTrigger	ALL
         resources.pipelines.*.trigger.cron.quartz_cron_schedule	string	ALL
TestAccept/bundle/resources/pipelines/testing/DMS=: branch ✅ | main (cec3f63) ❌ | latest (v1.26.0) ❌

main (cec3f63):

acceptance_test.go:1345: 
        	Error Trace:	/home/runner/work/cli/cli/libs/testdiff/testdiff.go:22
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1345
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1086
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:660
        	Error:      	Not equal: 
        	            	expected: "\n>>> [CLI] bundle validate -o json\n{\n  \"environment\": {\n    \"dependencies\": [\n      \"-e .[test]\",\n      \"-r ./tests/requirements.txt\",\n      \"./dist/*.whl\"\n    ]\n  },\n  \"sources\": [\n    {\n      \"glob\": {\n        \"include\": \"./tests/**\"\n      }\n    }\n  ]\n}\n\n=== Without pipeline testing, the property is an error\n\n>>> musterr env -u DATABRICKS_BUNDLE_ENABLE_EXPERIMENTAL_PIPELINE_TESTING [CLI] bundle validate\nError: the testing property of pipelines is experimental. Set DATABRICKS_BUNDLE_ENABLE_EXPERIMENTAL_PIPELINE_TESTING=true to enable it\n  at resources.pipelines.orders.testing\n  in databricks.yml:12:9\n\nName: pipeline-testing\nTarget: default\nWorkspace:\n  User: [USERNAME]\n  Path: /Workspace/Users/[USERNAME]/.bundle/pipeline-testing/default\n\nFound 1 error\n"
        	            	actual  : "\n>>> [CLI] bundle validate -o json\nWarning: unknown field: testing\n  at resources.pipelines.orders\n  in databricks.yml:11:7\n\nnull\n\n=== Without pipeline testing, the property is an error\n\n>>> musterr env -u DATABRICKS_BUNDLE_ENABLE_EXPERIMENTAL_PIPELINE_TESTING [CLI] bundle validate\nWarning: unknown field: testing\n  at resources.pipelines.orders\n  in databricks.yml:11:7\n\nName: pipeline-testing\nTarget: default\nWorkspace:\n  User: [USERNAME]\n  Path: /Workspace/Users/[USERNAME]/.bundle/pipeline-testing/default\n\nFound 1 warning\n\nUnexpected success\n\nExit code: 1\n"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -2,18 +2,7 @@
        	            	 >>> [CLI] bundle validate -o json
        	            	-{
        	            	-  "environment": {
        	            	-    "dependencies": [
        	            	-      "-e .[test]",
        	            	-      "-r ./tests/requirements.txt",
        	            	-      "./dist/*.whl"
        	            	-    ]
        	            	-  },
        	            	-  "sources": [
        	            	-    {
        	            	-      "glob": {
        	            	-        "include": "./tests/**"
        	            	-      }
        	            	-    }
        	            	-  ]
        	            	-}
        	            	+Warning: unknown field: testing
        	            	+  at resources.pipelines.orders
        	            	+  in databricks.yml:11:7
        	            	+
        	            	+null
        	            	 
        	            	@@ -22,5 +11,5 @@
        	            	 >>> musterr env -u DATABRICKS_BUNDLE_ENABLE_EXPERIMENTAL_PIPELINE_TESTING [CLI] bundle validate
        	            	-Error: the testing property of pipelines is experimental. Set DATABRICKS_BUNDLE_ENABLE_EXPERIMENTAL_PIPELINE_TESTING=true to enable it
        	            	-  at resources.pipelines.orders.testing
        	            	-  in databricks.yml:12:9
        	            	+Warning: unknown field: testing
        	            	+  at resources.pipelines.orders
        	            	+  in databricks.yml:11:7
        	            	 
        	            	@@ -32,3 +21,7 @@
        	            	 
        	            	-Found 1 error
        	            	+Found 1 warning
        	            	 
        	            	+Unexpected success
        	            	+
        	            	+Exit code: 1
        	            	+
        	Test:       	TestAccept/bundle/resources/pipelines/testing/DMS=
        	Messages:   	bundle/resources/pipelines/testing/output.txt vs /tmp/TestAcceptbundleresourcespipelinestestingDMS=3752144155/002/output.txt

latest (v1.26.0): same output as main (cec3f63)

TestAccept/bundle/resources/pipelines/testing/DMS=true: branch ✅ | main (cec3f63) ❌ | latest (v1.26.0) ❌

main (cec3f63):

acceptance_test.go:1345: 
        	Error Trace:	/home/runner/work/cli/cli/libs/testdiff/testdiff.go:22
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1345
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1086
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:660
        	Error:      	Not equal: 
        	            	expected: "\n>>> [CLI] bundle validate -o json\n{\n  \"environment\": {\n    \"dependencies\": [\n      \"-e .[test]\",\n      \"-r ./tests/requirements.txt\",\n      \"./dist/*.whl\"\n    ]\n  },\n  \"sources\": [\n    {\n      \"glob\": {\n        \"include\": \"./tests/**\"\n      }\n    }\n  ]\n}\n\n=== Without pipeline testing, the property is an error\n\n>>> musterr env -u DATABRICKS_BUNDLE_ENABLE_EXPERIMENTAL_PIPELINE_TESTING [CLI] bundle validate\nError: the testing property of pipelines is experimental. Set DATABRICKS_BUNDLE_ENABLE_EXPERIMENTAL_PIPELINE_TESTING=true to enable it\n  at resources.pipelines.orders.testing\n  in databricks.yml:12:9\n\nName: pipeline-testing\nTarget: default\nWorkspace:\n  User: [USERNAME]\n  Path: /Workspace/Users/[USERNAME]/.bundle/pipeline-testing/default\n\nFound 1 error\n"
        	            	actual  : "\n>>> [CLI] bundle validate -o json\nWarning: unknown field: testing\n  at resources.pipelines.orders\n  in databricks.yml:11:7\n\nnull\n\n=== Without pipeline testing, the property is an error\n\n>>> musterr env -u DATABRICKS_BUNDLE_ENABLE_EXPERIMENTAL_PIPELINE_TESTING [CLI] bundle validate\nWarning: unknown field: testing\n  at resources.pipelines.orders\n  in databricks.yml:11:7\n\nName: pipeline-testing\nTarget: default\nWorkspace:\n  User: [USERNAME]\n  Path: /Workspace/Users/[USERNAME]/.bundle/pipeline-testing/default\n\nFound 1 warning\n\nUnexpected success\n\nExit code: 1\n"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -2,18 +2,7 @@
        	            	 >>> [CLI] bundle validate -o json
        	            	-{
        	            	-  "environment": {
        	            	-    "dependencies": [
        	            	-      "-e .[test]",
        	            	-      "-r ./tests/requirements.txt",
        	            	-      "./dist/*.whl"
        	            	-    ]
        	            	-  },
        	            	-  "sources": [
        	            	-    {
        	            	-      "glob": {
        	            	-        "include": "./tests/**"
        	            	-      }
        	            	-    }
        	            	-  ]
        	            	-}
        	            	+Warning: unknown field: testing
        	            	+  at resources.pipelines.orders
        	            	+  in databricks.yml:11:7
        	            	+
        	            	+null
        	            	 
        	            	@@ -22,5 +11,5 @@
        	            	 >>> musterr env -u DATABRICKS_BUNDLE_ENABLE_EXPERIMENTAL_PIPELINE_TESTING [CLI] bundle validate
        	            	-Error: the testing property of pipelines is experimental. Set DATABRICKS_BUNDLE_ENABLE_EXPERIMENTAL_PIPELINE_TESTING=true to enable it
        	            	-  at resources.pipelines.orders.testing
        	            	-  in databricks.yml:12:9
        	            	+Warning: unknown field: testing
        	            	+  at resources.pipelines.orders
        	            	+  in databricks.yml:11:7
        	            	 
        	            	@@ -32,3 +21,7 @@
        	            	 
        	            	-Found 1 error
        	            	+Found 1 warning
        	            	 
        	            	+Unexpected success
        	            	+
        	            	+Exit code: 1
        	            	+
        	Test:       	TestAccept/bundle/resources/pipelines/testing/DMS=true
        	Messages:   	bundle/resources/pipelines/testing/output.txt vs /tmp/TestAcceptbundleresourcespipelinestestingDMS=true3935158443/002/output.txt

latest (v1.26.0):

acceptance_test.go:1345: 
        	Error Trace:	/home/runner/work/cli/cli/libs/testdiff/testdiff.go:22
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1345
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1086
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:660
        	Error:      	Not equal: 
        	            	expected: "\n>>> [CLI] bundle validate -o json\n{\n  \"environment\": {\n    \"dependencies\": [\n      \"-e .[test]\",\n      \"-r ./tests/requirements.txt\",\n      \"./dist/*.whl\"\n    ]\n  },\n  \"sources\": [\n    {\n      \"glob\": {\n        \"include\": \"./tests/**\"\n      }\n    }\n  ]\n}\n\n=== Without pipeline testing, the property is an error\n\n>>> musterr env -u DATABRICKS_BUNDLE_ENABLE_EXPERIMENTAL_PIPELINE_TESTING [CLI] bundle validate\nError: the testing property of pipelines is experimental. Set DATABRICKS_BUNDLE_ENABLE_EXPERIMENTAL_PIPELINE_TESTING=true to enable it\n  at resources.pipelines.orders.testing\n  in databricks.yml:12:9\n\nName: pipeline-testing\nTarget: default\nWorkspace:\n  User: [USERNAME]\n  Path: /Workspace/Users/[USERNAME]/.bundle/pipeline-testing/default\n\nFound 1 error\n"
        	            	actual  : "\n>>> [CLI] bundle validate -o json\nWarning: unknown field: testing\n  at resources.pipelines.orders\n  in databricks.yml:11:7\n\nnull\n\n=== Without pipeline testing, the property is an error\n\n>>> musterr env -u DATABRICKS_BUNDLE_ENABLE_EXPERIMENTAL_PIPELINE_TESTING [CLI] bundle validate\nWarning: unknown field: testing\n  at resources.pipelines.orders\n  in databricks.yml:11:7\n\nName: pipeline-testing\nTarget: default\nWorkspace:\n  User: [USERNAME]\n  Path: /Workspace/Users/[USERNAME]/.bundle/pipeline-testing/default\n\nFound 1 warning\n\nUnexpected success\n\nExit code: 1\n"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -2,18 +2,7 @@
        	            	 >>> [CLI] bundle validate -o json
        	            	-{
        	            	-  "environment": {
        	            	-    "dependencies": [
        	            	-      "-e .[test]",
        	            	-      "-r ./tests/requirements.txt",
        	            	-      "./dist/*.whl"
        	            	-    ]
        	            	-  },
        	            	-  "sources": [
        	            	-    {
        	            	-      "glob": {
        	            	-        "include": "./tests/**"
        	            	-      }
        	            	-    }
        	            	-  ]
        	            	-}
        	            	+Warning: unknown field: testing
        	            	+  at resources.pipelines.orders
        	            	+  in databricks.yml:11:7
        	            	+
        	            	+null
        	            	 
        	            	@@ -22,5 +11,5 @@
        	            	 >>> musterr env -u DATABRICKS_BUNDLE_ENABLE_EXPERIMENTAL_PIPELINE_TESTING [CLI] bundle validate
        	            	-Error: the testing property of pipelines is experimental. Set DATABRICKS_BUNDLE_ENABLE_EXPERIMENTAL_PIPELINE_TESTING=true to enable it
        	            	-  at resources.pipelines.orders.testing
        	            	-  in databricks.yml:12:9
        	            	+Warning: unknown field: testing
        	            	+  at resources.pipelines.orders
        	            	+  in databricks.yml:11:7
        	            	 
        	            	@@ -32,3 +21,7 @@
        	            	 
        	            	-Found 1 error
        	            	+Found 1 warning
        	            	 
        	            	+Unexpected success
        	            	+
        	            	+Exit code: 1
        	            	+
        	Test:       	TestAccept/bundle/resources/pipelines/testing/DMS=true
        	Messages:   	bundle/resources/pipelines/testing/output.txt vs /tmp/TestAcceptbundleresourcespipelinestestingDMS=true904311867/002/output.txt
FAIL

@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Nov 2, 2026 •

Copy link
Copy Markdown
Collaborator

✅ Integration test report

Commit: 86997dc

Run: 43108389234

Env 🔄​flaky 💚​RECOVERED ✅​pass 🙈​skip Time
💚​ aws linux-2core-8gb 1 270 18 8:03
🔄​ aws-windows-latest-4core-16gb 2 1 270 16 6:30
💚​ azure linux-2core-8gb 1 269 18 8:27
💚​ azure-windows-latest-4core-16gb 1 271 16 5:39
💚​ gcp linux-2core-8gb 1 270 18 9:16
💚​ gcp-windows-latest-4core-16gb 1 272 16 6:24
Test Name aws linux-2core-8gb aws-windows-latest-4core-16gb azure linux-2core-8gb azure-windows-latest-4core-16gb gcp linux-2core-8gb gcp-windows-latest-4core-16gb
💚​ TestAccept 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
🔄​ TestFsCpFileToFileFileNotOverwritten ✅​p 🔄​f ✅​p ✅​p ✅​p ✅​p
🔄​ TestFsCpFileToFileFileNotOverwritten/dbfs_to_local ✅​p 🔄​f ✅​p ✅​p ✅​p ✅​p
Top 7 slowest tests (at least 2 minutes):
duration env testname
3:42 aws-windows-latest-4core-16gb TestAccept
3:39 azure-windows-latest-4core-16gb TestAccept
3:39 gcp-windows-latest-4core-16gb TestAccept
3:20 gcp linux-2core-8gb TestAccept
3:16 aws linux-2core-8gb TestAccept
3:08 azure linux-2core-8gb TestAccept
2:05 azure-windows-latest-4core-16gb TestFilerReadWrite/workspace_files_extensions

@tharunbhuk-db
tharunbhuk-db force-pushed the tharun/pipelines-test-pr3-junit branch from cc2e61a to b4df318 Compare November 3, 2026 00:53
@tharunbhuk-db
tharunbhuk-db force-pushed the tharun/pipelines-test-pr4-bundles branch from 9c9d9c8 to 742ef87 Compare November 3, 2026 00:53
@tharunbhuk-db
tharunbhuk-db force-pushed the tharun/pipelines-test-pr4-bundles branch from e038a13 to e895d35 Compare November 3, 2026 17:34
@tharunbhuk-db
tharunbhuk-db force-pushed the tharun/pipelines-test-pr3-junit branch from b4df318 to 7c61f68 Compare November 3, 2026 17:34
Comment thread bundle/direct/dresources/pipeline.go Resolved
Comment thread bundle/internal/schema/annotations.yml Resolved
Comment thread libs/testserver/pipelines.go Resolved
@tharunbhuk-db
tharunbhuk-db marked this pull request as ready for review November 5, 2026 16:09
@tharunbhuk-db
tharunbhuk-db marked this pull request as draft November 9, 2026 23:57
@tharunbhuk-db tharunbhuk-db changed the title [WIP] Add the testing property of pipelines to bundles Add the testing property of pipelines to bundles Nov 10, 2026
@tharunbhuk-db
tharunbhuk-db changed the base branch from tharun/pipelines-test-pr3-junit to tharun/pipelines-test-b1-editable-folder November 10, 2026 00:25
@tharunbhuk-db
tharunbhuk-db force-pushed the tharun/pipelines-test-b1-editable-folder branch from abc52a2 to 6092358 Compare November 10, 2026 00:25
@tharunbhuk-db
tharunbhuk-db force-pushed the tharun/pipelines-test-pr4-bundles branch from 286aa60 to 9791a45 Compare November 10, 2026 00:25
@tharunbhuk-db
tharunbhuk-db force-pushed the tharun/pipelines-test-b1-editable-folder branch from 6092358 to 7e4c017 Compare November 10, 2026 10:06
@tharunbhuk-db
tharunbhuk-db force-pushed the tharun/pipelines-test-pr4-bundles branch from 9791a45 to 239501f Compare November 10, 2026 10:06
@tharunbhuk-db
tharunbhuk-db force-pushed the tharun/pipelines-test-b1-editable-folder branch from 7e4c017 to bf6225d Compare November 10, 2026 10:46
@tharunbhuk-db
tharunbhuk-db force-pushed the tharun/pipelines-test-pr4-bundles branch from 239501f to bc013c0 Compare November 10, 2026 10:46
@tharunbhuk-db
tharunbhuk-db force-pushed the tharun/pipelines-test-b1-editable-folder branch from bf6225d to e98a9f7 Compare November 10, 2026 16:49
@tharunbhuk-db
tharunbhuk-db force-pushed the tharun/pipelines-test-pr4-bundles branch from bc013c0 to 08916f8 Compare November 10, 2026 16:49
@tharunbhuk-db
tharunbhuk-db force-pushed the tharun/pipelines-test-b1-editable-folder branch from e98a9f7 to 9a2cc3c Compare November 10, 2026 17:22
@tharunbhuk-db
tharunbhuk-db force-pushed the tharun/pipelines-test-pr4-bundles branch from 08916f8 to 4c21f8b Compare November 10, 2026 17:22
@tharunbhuk-db

Copy link
Copy Markdown
Contributor Author

Follow-ups from the review:

@tharunbhuk-db
tharunbhuk-db marked this pull request as ready for review November 10, 2026 17:43

@pranshupand-db pranshupand-db left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Tharun. Splitting the -e mapping and the deploy path into their own PRs made this much easier to check. LGTM for Stack B (#7427 → #7300 → #7437 → #7438 → #7429 → #7439).

What I checked for "doesn't change existing behavior":

  • Code. The new paths only run when testing is set or DATABRICKS_BUNDLE_ENABLE_EXPERIMENTAL_PIPELINE_TESTING is on. The new path patterns only match testing.*, the resource field is omitempty, and outside the feature only the refschema and help goldens change.
  • Live on dogfood staging. I built the stack top (5b1b0b0) and its main base. For a bundle with a plain pipeline and a job deployed by main, Stack B's plan and deploy showed 0 changes with the variable off and on, bundle validate -o json was identical to main, and main read the state Stack B wrote with 0 changes. A plain pipeline's Get has no testing key, so there's no spurious drift.
  • The testing lifecycle. Adding testing and deploying sets it, and the next plan shows 0 changes. Removing it from YAML and deploying clears spec.testing on the server, which answers the open PUT question from the earlier review. testing set outside the bundle shows as drift and is undone with the variable on, and is untouched with it off, as the changelog says.

Notes (not blocking):

  1. Mixed CLI versions. A released CLI only warns unknown field: testing, and a deploy from it that changes the pipeline clears spec.testing (verified). That's how PUT works, not this stack, but let's say in the release note or docs that everyone deploying the bundle needs the new CLI with the variable set.
  2. Public text. The schema description for testing (annotations.yml, which feeds the JSON schema and Python docstrings) says "[Private Preview]". If that label shouldn't reach customers yet, let's reword it.
  3. Follow-up. Map --editable ./project[...] like -e (your item from the earlier thread).

}
var diags diag.Diagnostics
pipelines := b.Config.Resources.Pipelines
for _, key := range slices.Sorted(maps.Keys(pipelines)) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pipeline_development_deprecated.go right above this in initialize.go does the same walk with dyn.MapByPattern on resources.pipelines.*.development, which gives the path and locations directly. Can we do the same here for resources.pipelines.*.testing instead of iterating the typed map and calling dyn.GetByPath per key?

"github.com/stretchr/testify/require"
)

func TestPipelineTestingEnabled(t *testing.T) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need this unit test? The acceptance test already shows the error with its location and path (the env -u ... bundle validate step) and the enabled case, and we try not to unit test mutators that an acceptance output covers.

"file_path",
},
"pipelines": {
// The experimental testing property is not deployed yet.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If this lands before #7438, a bundle with the variable on and testing set deploys fine and testing is silently dropped (PrepareState copies only CreatePipeline and CascadeOnDestroy). Fine for an experimental flag while the stack merges together, or do we want validate to say the property isn't deployed yet?

@tharunbhuk-db
tharunbhuk-db force-pushed the tharun/pipelines-test-pr4-bundles branch from 4c21f8b to 33085b7 Compare November 11, 2026 19:58
@tharunbhuk-db
tharunbhuk-db marked this pull request as draft November 11, 2026 20:01
Add resources.pipelines.<key>.testing, with the test sources and test-only
dependencies that databricks pipelines test runs with. The CLI has its own
types for it, because the Go SDK does not have the field yet.

The property is experimental. It needs
DATABRICKS_BUNDLE_ENABLE_EXPERIMENTAL_PIPELINE_TESTING=true. While the variable
is off, bundle validate rejects testing and names the variable. Bundles without
testing are not changed. Later PRs translate its paths and deploy it.

Co-authored-by: Isaac <no-reply@databricks.com>
@tharunbhuk-db
tharunbhuk-db force-pushed the tharun/pipelines-test-pr4-bundles branch from 33085b7 to 86997dc Compare November 11, 2026 22:00
@tharunbhuk-db
tharunbhuk-db force-pushed the tharun/pipelines-test-b1-editable-folder branch from 2a9184c to 24212b4 Compare November 11, 2026 22:00

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants