Skip to content

Record deployment history for bundle resource bindings - #7400

Open
shreyas-goenka wants to merge 28 commits into
mainfrom
shreyas-goenka_data/shreyas-goenka/record-bind-history
Open

shreyas-goenka wants to merge 28 commits into
mainfrom
shreyas-goenka_data/shreyas-goenka/record-bind-history

Conversation

@shreyas-goenka

@shreyas-goenka shreyas-goenka commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

Changes

Record successful bundle deployment bind commands as UPDATE operations in deployment history. Complete the history version after remote state upload; history failures warn while state upload continues.

Use the existing dms.OperationBuffer, which StateDB also uses, for state validation, uploads, bounded best-effort draining, and heartbeats. The engine returns optional DMS metadata with the committed resource and serial. No extra state writes are added, and existing deployment metadata is preserved because bind does not upload a snapshot.

Direct StateDB recording was evaluated: bind's current SaveState calls prepare temporary state before approval. Sharing that path would require moving the approved commit into StateDB, or adding a second state write that advances the serial. Reusing its DMS buffer removes the duplicate operation-update code while preserving the binding flow.

Shared deployment lookup lives in libs/dms. First in a stack; #7401 adds unbind recording.

Tests

  • Focused bind/unbind, bind history failures, and immutable-folder acceptance cases pass.
  • Scoped Go lint passes; snapshots were regenerated.
  • Rebased on current main and resolved conflicts. Fresh platform and cloud CI is running.

This PR was written by Codex.

@github-actions github-actions Bot added the DABs DABs related issues label Nov 9, 2026
@shreyas-goenka
shreyas-goenka force-pushed the shreyas-goenka_data/shreyas-goenka/record-bind-history branch from c8357c9 to 938294b Compare November 9, 2026 07:59
@shreyas-goenka
shreyas-goenka force-pushed the shreyas-goenka_data/shreyas-goenka/record-bind-history branch from 938294b to 2c7cf79 Compare November 9, 2026 08:08
@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Nov 9, 2026 •

Copy link
Copy Markdown
Collaborator

✅ Integration test report

Commit: 7b569b3

Run: 43429751862

Env 💚​RECOVERED ✅​pass 🙈​skip Time
💚​ aws linux-2core-8gb 1 280 18 9:44
💚​ aws-windows-latest-4core-16gb 1 282 16 7:02
💚​ azure linux-2core-8gb 1 279 18 11:20
💚​ azure-windows-latest-4core-16gb 1 281 16 5:36
💚​ gcp linux-2core-8gb 1 280 18 11:59
💚​ gcp-windows-latest-4core-16gb 1 282 16 6:02
Test Name aws linux-2core-8gb aws-windows-latest-4core-16gb azure linux-2core-8gb azure-windows-latest-4core-16gb gcp linux-2core-8gb gcp-windows-latest-4core-16gb
💚​ TestAccept 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
Top 18 slowest tests (at least 2 minutes):
duration env testname
4:21 azure linux-2core-8gb TestAccept
4:01 gcp linux-2core-8gb TestAccept
3:55 aws linux-2core-8gb TestAccept
3:11 aws-windows-latest-4core-16gb TestAccept
3:01 gcp-windows-latest-4core-16gb TestAccept
2:59 azure-windows-latest-4core-16gb TestAccept
2:49 gcp linux-2core-8gb TestFilerWorkspaceFilesExtensionsDelete
2:35 aws-windows-latest-4core-16gb TestAccept/bundle/deployment/bind/dashboard/recreation/DMS=/READPLAN=
2:34 gcp-windows-latest-4core-16gb TestAccept/bundle/dms/bind-unbind/DMS=true
2:26 gcp linux-2core-8gb TestAccept/bundle/dms/bind-unbind/DMS=true
2:24 gcp linux-2core-8gb TestAccept/bundle/deployment/bind/dashboard/recreation/DMS=/READPLAN=1
2:16 aws-windows-latest-4core-16gb TestFilerWorkspaceFilesExtensionsRead
2:12 azure-windows-latest-4core-16gb TestAccept/bundle/deployment/bind/dashboard/recreation/DMS=true/READPLAN=
2:07 azure linux-2core-8gb TestAccept/bundle/deployment/bind/dashboard/recreation/DMS=/READPLAN=
2:04 aws linux-2core-8gb TestAccept/bundle/dms/bind-unbind/DMS=true
2:02 aws linux-2core-8gb TestFilerWorkspaceFilesExtensionsReadDir
2:01 gcp-windows-latest-4core-16gb TestAccept/bundle/deployment/bind/dashboard/recreation/DMS=/READPLAN=
2:00 azure-windows-latest-4core-16gb TestAccept/bundle/deployment/bind/dashboard/recreation/DMS=/READPLAN=

@github-actions

github-actions Bot commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

Commit: 7b569b3a5 | Workflow run

Regression test report

Selected 6 changed tests (limit=20, 0 not selected)

11 tests: regression=3 unreleased=0 coverage=8 failing=0
test branch main (5ed1897) latest (v1.26.0)
TestAccept/bundle/deployment/bind/dashboard/recreation/DMS=/READPLAN= ✅ ✅ ✅
TestAccept/bundle/deployment/bind/dashboard/recreation/DMS=/READPLAN=1 ✅ ✅ ✅
TestAccept/bundle/deployment/bind/dashboard/recreation/DMS=true/READPLAN= ✅ ✅ ✅
TestAccept/bundle/deployment/bind/dashboard/recreation/DMS=true/READPLAN=1 ✅ ✅ ✅
TestAccept/bundle/deployment/bind/pipelines/recreate/DMS= ✅ ✅ ✅
TestAccept/bundle/deployment/bind/pipelines/recreate/DMS=true ✅ ✅ ✅
TestAccept/bundle/deployment/bind/pipelines/update/DMS= ✅ ✅ ✅
TestAccept/bundle/deployment/bind/pipelines/update/DMS=true ✅ ✅ ✅
TestAccept/bundle/dms/bind-history-failures/DMS=true ✅ ❌ ❌
TestAccept/bundle/dms/bind-immutable/DMS=true ✅ ❌ ❌
TestAccept/bundle/dms/bind-unbind/DMS=true ✅ ❌ ❌
TestAccept/bundle/dms/bind-history-failures/DMS=true: branch ✅ | main (5ed1897) ❌ | latest (v1.26.0) ❌

main (5ed1897):

acceptance_test.go:1374: Diff:
        --- bundle/dms/bind-history-failures/output.txt
        +++ /tmp/TestAcceptbundledmsbind-history-failuresDMS=true3054889300/002/output.txt
        @@ -4,7 +4,6 @@
         >>> fault.py POST /api/2.0/bundle/deployments/*/versions 400 0 1
         
         >>> [CLI] bundle deployment bind create_failed [JOB_ID] --auto-approve
        -Warn: Failed to record bind history: failed to create deployment version: Fault injected by test.
         Successfully bound job with an id '[JOB_ID]'
         Run 'bundle deploy' to deploy changes to your workspace
         
        @@ -13,7 +12,6 @@
         >>> fault.py PATCH /api/2.0/bundle/** 400 0 1
         
         >>> [CLI] bundle deployment bind record_failed [JOB_ID] --auto-approve
        -Warn: Failed to record bind history: recording operation for resources.jobs.record_failed: Fault injected by test.
         Successfully bound job with an id '[JOB_ID]'
         Run 'bundle deploy' to deploy changes to your workspace
         
        @@ -22,7 +20,6 @@
         >>> fault.py POST /api/2.0/bundle/deployments/*/versions/*/complete 400 0 1
         
         >>> [CLI] bundle deployment bind complete_failed [JOB_ID] --auto-approve
        -Warn: Failed to complete deployment history version: Fault injected by test.
         Successfully bound job with an id '[JOB_ID]'
         Run 'bundle deploy' to deploy changes to your workspace
         
        @@ -42,13 +39,6 @@
         }
         
         >>> print_requests.py --dms //versions --oneline
        -{"method": "POST", "path": "/api/2.0/bundle/deployments/[DEPLOYMENT_ID]/versions", "q": {"version_id": "3"}, "body": {"cli_version": "[CLI_VERSION]", "operations": [{"action_type": "OPERATION_ACTION_TYPE_UPDATE", "resource_key": "jobs.create_failed"}], "version_type": "VERSION_TYPE_DEPLOY"}}
        -{"method": "POST", "path": "/api/2.0/bundle/deployments/[DEPLOYMENT_ID]/versions", "q": {"version_id": "5"}, "body": {"cli_version": "[CLI_VERSION]", "operations": [{"action_type": "OPERATION_ACTION_TYPE_UPDATE", "resource_key": "jobs.record_failed"}], "version_type": "VERSION_TYPE_DEPLOY"}}
        -{"method": "PATCH", "path": "/api/2.0/bundle/deployments/[DEPLOYMENT_ID]/versions/5/operations/jobs.record_failed", "q": {"update_mask": "state,error_message,resource_id,status"}, "body": {"error_message": "", "resource_id": "[JOB_ID]", "sequence_id": 0, "state": "{\"state\":{\"deployment\":{\"kind\":\"BUNDLE\",\"metadata_file_path\":\"/Workspace/Users/[USERNAME]/.bundle/dms-bind-history-failures-[UNIQUE_NAME]/default/state/metadata.json\"},\"edit_mode\":\"UI_LOCKED\",\"format\":\"MULTI_TASK\",\"max_concurrent_runs\":1,\"name\":\"record_failed\",\"queue\":{\"enabled\":true}}}", "status": "OPERATION_STATUS_SUCCEEDED"}}
        -{"method": "POST", "path": "/api/2.0/bundle/deployments/[DEPLOYMENT_ID]/versions/5/complete", "body": {"completion_reason": "VERSION_COMPLETE_FAILURE"}}
        -{"method": "POST", "path": "/api/2.0/bundle/deployments/[DEPLOYMENT_ID]/versions", "q": {"version_id": "7"}, "body": {"cli_version": "[CLI_VERSION]", "operations": [{"action_type": "OPERATION_ACTION_TYPE_UPDATE", "resource_key": "jobs.complete_failed"}], "previous_version_id": "5", "version_type": "VERSION_TYPE_DEPLOY"}}
        -{"method": "PATCH", "path": "/api/2.0/bundle/deployments/[DEPLOYMENT_ID]/versions/7/operations/jobs.complete_failed", "q": {"update_mask": "state,error_message,resource_id,status"}, "body": {"error_message": "", "resource_id": "[JOB_ID]", "sequence_id": 0, "state": "{\"state\":{\"deployment\":{\"kind\":\"BUNDLE\",\"metadata_file_path\":\"/Workspace/Users/[USERNAME]/.bundle/dms-bind-history-failures-[UNIQUE_NAME]/default/state/metadata.json\"},\"edit_mode\":\"UI_LOCKED\",\"format\":\"MULTI_TASK\",\"max_concurrent_runs\":1,\"name\":\"complete_failed\",\"queue\":{\"enabled\":true}}}", "status": "OPERATION_STATUS_SUCCEEDED"}}
        -{"method": "POST", "path": "/api/2.0/bundle/deployments/[DEPLOYMENT_ID]/versions/7/complete", "body": {"completion_reason": "VERSION_COMPLETE_SUCCESS"}}
         
         === A remote state upload failure fails the bind version
         
        @@ -64,9 +54,6 @@
         
         
         >>> print_requests.py --dms //versions --oneline
        -{"method": "POST", "path": "/api/2.0/bundle/deployments/[DEPLOYMENT_ID]/versions", "q": {"version_id": "9"}, "body": {"cli_version": "[CLI_VERSION]", "operations": [{"action_type": "OPERATION_ACTION_TYPE_UPDATE", "resource_key": "jobs.upload_failed"}], "previous_version_id": "7", "version_type": "VERSION_TYPE_DEPLOY"}}
        -{"method": "PATCH", "path": "/api/2.0/bundle/deployments/[DEPLOYMENT_ID]/versions/9/operations/jobs.upload_failed", "q": {"update_mask": "state,error_message,resource_id,status"}, "body": {"error_message": "", "resource_id": "[JOB_ID]", "sequence_id": 0, "state": "{\"state\":{\"deployment\":{\"kind\":\"BUNDLE\",\"metadata_file_path\":\"/Workspace/Users/[USERNAME]/.bundle/dms-bind-history-failures-[UNIQUE_NAME]/default/state/metadata.json\"},\"edit_mode\":\"UI_LOCKED\",\"format\":\"MULTI_TASK\",\"max_concurrent_runs\":1,\"name\":\"upload_failed\",\"queue\":{\"enabled\":true}}}", "status": "OPERATION_STATUS_SUCCEEDED"}}
        -{"method": "POST", "path": "/api/2.0/bundle/deployments/[DEPLOYMENT_ID]/versions/9/complete", "body": {"completion_reason": "VERSION_COMPLETE_FAILURE"}}
         
         === Remote state still contains only the successful bindings

latest (v1.26.0): same output as main (5ed1897)

TestAccept/bundle/dms/bind-immutable/DMS=true: branch ✅ | main (5ed1897) ❌ | latest (v1.26.0) ❌

main (5ed1897):

acceptance_test.go:1374: 
        	Error Trace:	/home/runner/work/cli/cli/libs/testdiff/testdiff.go:22
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1374
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1115
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:678
        	Error:      	Not equal: 
        	            	expected: "\n>>> [CLI] bundle deployment bind foo [JOB_ID] --auto-approve\nSuccessfully bound job with an id '[JOB_ID]'\nRun 'bundle deploy' to deploy changes to your workspace\n\n>>> MSYS_NO_PATHCONV=1 [CLI] bundle-deployments list-resources deployments/[DEPLOYMENT_ID] -o json\n[\n  {\n    \"last_action_type\": \"OPERATION_ACTION_TYPE_UPDATE\",\n    \"resource_id\": \"[JOB_ID]\",\n    \"resource_key\": \"jobs.foo\"\n  }\n]\n"
        	            	actual  : "\n>>> [CLI] bundle deployment bind foo [JOB_ID] --auto-approve\nSuccessfully bound job with an id '[JOB_ID]'\nRun 'bundle deploy' to deploy changes to your workspace\nError: Path (/Workspace/Users/[USERNAME]/.bundle/dms-bind-immutable-[UNIQUE_NAME]/default/state/resources.deployment.json) doesn't exist.\n\nExit code: 1\n"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -4,11 +4,5 @@
        	            	 Run 'bundle deploy' to deploy changes to your workspace
        	            	+Error: Path (/Workspace/Users/[USERNAME]/.bundle/dms-bind-immutable-[UNIQUE_NAME]/default/state/resources.deployment.json) doesn't exist.
        	            	 
        	            	->>> MSYS_NO_PATHCONV=1 [CLI] bundle-deployments list-resources deployments/[DEPLOYMENT_ID] -o json
        	            	-[
        	            	-  {
        	            	-    "last_action_type": "OPERATION_ACTION_TYPE_UPDATE",
        	            	-    "resource_id": "[JOB_ID]",
        	            	-    "resource_key": "jobs.foo"
        	            	-  }
        	            	-]
        	            	+Exit code: 1
        	            	 
        	Test:       	TestAccept/bundle/dms/bind-immutable/DMS=true
        	Messages:   	bundle/dms/bind-immutable/output.txt vs /tmp/TestAcceptbundledmsbind-immutableDMS=true200248722/002/output.txt

latest (v1.26.0): same output as main (5ed1897)

TestAccept/bundle/dms/bind-unbind/DMS=true: branch ✅ | main (5ed1897) ❌ | latest (v1.26.0) ❌

main (5ed1897):

acceptance_test.go:1374: Diff:
        --- bundle/dms/bind-unbind/output.txt
        +++ /tmp/TestAcceptbundledmsbind-unbindDMS=true3817926208/002/output.txt
        @@ -20,98 +20,12 @@
         Run 'bundle deploy' to deploy changes to your workspace
         
         >>> print_requests.py --dms //versions --oneline
        -{"method": "POST", "path": "/api/2.0/bundle/deployments/[DEPLOYMENT_ID]/versions", "q": {"version_id": "3"}, "body": {"cli_version": "[CLI_VERSION]", "operations": [{"action_type": "OPERATION_ACTION_TYPE_UPDATE", "resource_key": "jobs.foo"}], "version_type": "VERSION_TYPE_DEPLOY"}}
        -{"method": "PATCH", "path": "/api/2.0/bundle/deployments/[DEPLOYMENT_ID]/versions/3/operations/jobs.foo", "q": {"update_mask": "state,error_message,resource_id,status"}, "body": {"error_message": "", "resource_id": "[JOB_ID]", "sequence_id": 0, "state": "{\"state\":{\"deployment\":{\"kind\":\"BUNDLE\",\"metadata_file_path\":\"/Workspace/Users/[USERNAME]/.bundle/dms-bind-unbind-[UNIQUE_NAME]/default/state/metadata.json\"},\"edit_mode\":\"UI_LOCKED\",\"format\":\"MULTI_TASK\",\"max_concurrent_runs\":1,\"name\":\"foo\",\"queue\":{\"enabled\":true}}}", "status": "OPERATION_STATUS_SUCCEEDED"}}
        -{"method": "POST", "path": "/api/2.0/bundle/deployments/[DEPLOYMENT_ID]/versions/3/complete", "body": {"completion_reason": "VERSION_COMPLETE_SUCCESS"}}
         
         >>> [CLI] jobs get [JOB_ID]
         {
           "job_id": [JOB_ID],
           "name": "bind-target-[UNIQUE_NAME]"
         }
        +Error: Path (/Workspace/Users/[USERNAME]/.bundle/dms-bind-unbind-[UNIQUE_NAME]/default/state/resources.deployment.json) doesn't exist.
         
        ->>> MSYS_NO_PATHCONV=1 [CLI] bundle-deployments list-resources deployments/[DEPLOYMENT_ID] -o json
        -[
        -  {
        -    "last_action_type": "OPERATION_ACTION_TYPE_UPDATE",
        -    "resource_id": "[JOB_ID]",
        -    "resource_key": "jobs.foo"
        -  }
        -]
        -
        -=== Deploy keeps the bound job ID with recorded history
        -
        ->>> [CLI] bundle deploy
        -Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-bind-unbind-[UNIQUE_NAME]/default/files...
        -Updated jobs.foo
        -Files: 2 uploaded, 0 deleted
        -Resources: 0 created, 1 changed, 0 deleted, 0 unchanged
        -
        ->>> jq {features, resource_id: .state["resources.jobs.foo"].__id__} .databricks/bundle/default/resources.json
        -{
        -  "features": {
        -    "deployment_history_shadow": {}
        -  },
        -  "resource_id": "[JOB_ID]"
        -}
        -
        -=== Unbind preserves the shadow history marker and leaves the job intact
        -
        ->>> [CLI] bundle deployment unbind foo
        -
        ->>> jq {features, state} .databricks/bundle/default/resources.json
        -{
        -  "features": {
        -    "deployment_history_shadow": {}
        -  },
        -  "state": {}
        -}
        -
        ->>> [CLI] jobs get [JOB_ID]
        -{
        -  "job_id": [JOB_ID]
        -}
        -
        -=== Rebind preserves the existing shadow history marker
        -
        ->>> [CLI] bundle deployment bind foo [JOB_ID] --auto-approve
        -Successfully bound job with an id '[JOB_ID]'
        -Run 'bundle deploy' to deploy changes to your workspace
        -
        ->>> jq {features, resource_id: .state["resources.jobs.foo"].__id__} .databricks/bundle/default/resources.json
        -{
        -  "features": {
        -    "deployment_history_shadow": {}
        -  },
        -  "resource_id": "[JOB_ID]"
        -}
        -
        ->>> print_requests.py --dms //versions --oneline
        -{"method": "POST", "path": "/api/2.0/bundle/deployments/[DEPLOYMENT_ID]/versions", "q": {"version_id": "7"}, "body": {"cli_version": "[CLI_VERSION]", "operations": [{"action_type": "OPERATION_ACTION_TYPE_UPDATE", "resource_key": "jobs.foo"}], "previous_version_id": "4", "version_type": "VERSION_TYPE_DEPLOY"}}
        -{"method": "PATCH", "path": "/api/2.0/bundle/deployments/[DEPLOYMENT_ID]/versions/7/operations/jobs.foo", "q": {"update_mask": "state,error_message,resource_id,status"}, "body": {"error_message": "", "resource_id": "[JOB_ID]", "sequence_id": 0, "state": "{\"state\":{\"deployment\":{\"kind\":\"BUNDLE\",\"metadata_file_path\":\"/Workspace/Users/[USERNAME]/.bundle/dms-bind-unbind-[UNIQUE_NAME]/default/state/metadata.json\"},\"edit_mode\":\"UI_LOCKED\",\"format\":\"MULTI_TASK\",\"max_concurrent_runs\":1,\"name\":\"foo\",\"queue\":{\"enabled\":true}}}", "status": "OPERATION_STATUS_SUCCEEDED"}}
        -{"method": "POST", "path": "/api/2.0/bundle/deployments/[DEPLOYMENT_ID]/versions/7/complete", "body": {"completion_reason": "VERSION_COMPLETE_SUCCESS"}}
        -
        -=== A subsequent deploy continues managing the same job
        -
        ->>> [CLI] bundle deploy
        -Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-bind-unbind-[UNIQUE_NAME]/default/files...
        -Files: 0 uploaded, 0 deleted
        -Resources: 0 created, 0 changed, 0 deleted, 1 unchanged
        -
        ->>> jq {features, resource_id: .state["resources.jobs.foo"].__id__} .databricks/bundle/default/resources.json
        -{
        -  "features": {
        -    "deployment_history_shadow": {}
        -  },
        -  "resource_id": "[JOB_ID]"
        -}
        -
        -=== A rejected duplicate bind creates no history version
        -
        ->>> musterr [CLI] bundle deployment bind foo [JOB_ID] --auto-approve
        -Error: Resource already managed
        -
        -The bundle is already managing resource for resources.jobs.foo and it is bound to the requested ID [JOB_ID].
        -
        -
        ->>> print_requests.py --dms //versions
        +Exit code: 1
        
    acceptance_test.go:1152: 
        LOG.cleanup: The following resources will be deleted:
        LOG.cleanup:   delete resources.jobs.foo
        LOG.cleanup: 
        LOG.cleanup: All files and directories at the following location will be deleted: /Workspace/Users/tester@databricks.com/.bundle/dms-bind-unbind-xcozaf4nqfcc7bvfikkji76awa/default
        LOG.cleanup: 
        LOG.cleanup: Destroy: 1 deleted
        LOG.cleanup: Error: Not Found
        LOG.cleanup: 
        LOG.cleanup: Exit code: 1
FAIL

latest (v1.26.0):

acceptance_test.go:1374: Diff:
        --- bundle/dms/bind-unbind/output.txt
        +++ /tmp/TestAcceptbundledmsbind-unbindDMS=true4270163812/002/output.txt
        @@ -20,98 +20,12 @@
         Run 'bundle deploy' to deploy changes to your workspace
         
         >>> print_requests.py --dms //versions --oneline
        -{"method": "POST", "path": "/api/2.0/bundle/deployments/[DEPLOYMENT_ID]/versions", "q": {"version_id": "3"}, "body": {"cli_version": "[CLI_VERSION]", "operations": [{"action_type": "OPERATION_ACTION_TYPE_UPDATE", "resource_key": "jobs.foo"}], "version_type": "VERSION_TYPE_DEPLOY"}}
        -{"method": "PATCH", "path": "/api/2.0/bundle/deployments/[DEPLOYMENT_ID]/versions/3/operations/jobs.foo", "q": {"update_mask": "state,error_message,resource_id,status"}, "body": {"error_message": "", "resource_id": "[JOB_ID]", "sequence_id": 0, "state": "{\"state\":{\"deployment\":{\"kind\":\"BUNDLE\",\"metadata_file_path\":\"/Workspace/Users/[USERNAME]/.bundle/dms-bind-unbind-[UNIQUE_NAME]/default/state/metadata.json\"},\"edit_mode\":\"UI_LOCKED\",\"format\":\"MULTI_TASK\",\"max_concurrent_runs\":1,\"name\":\"foo\",\"queue\":{\"enabled\":true}}}", "status": "OPERATION_STATUS_SUCCEEDED"}}
        -{"method": "POST", "path": "/api/2.0/bundle/deployments/[DEPLOYMENT_ID]/versions/3/complete", "body": {"completion_reason": "VERSION_COMPLETE_SUCCESS"}}
         
         >>> [CLI] jobs get [JOB_ID]
         {
           "job_id": [JOB_ID],
           "name": "bind-target-[UNIQUE_NAME]"
         }
        +Error: Path (/Workspace/Users/[USERNAME]/.bundle/dms-bind-unbind-[UNIQUE_NAME]/default/state/resources.deployment.json) doesn't exist.
         
        ->>> MSYS_NO_PATHCONV=1 [CLI] bundle-deployments list-resources deployments/[DEPLOYMENT_ID] -o json
        -[
        -  {
        -    "last_action_type": "OPERATION_ACTION_TYPE_UPDATE",
        -    "resource_id": "[JOB_ID]",
        -    "resource_key": "jobs.foo"
        -  }
        -]
        -
        -=== Deploy keeps the bound job ID with recorded history
        -
        ->>> [CLI] bundle deploy
        -Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-bind-unbind-[UNIQUE_NAME]/default/files...
        -Updated jobs.foo
        -Files: 2 uploaded, 0 deleted
        -Resources: 0 created, 1 changed, 0 deleted, 0 unchanged
        -
        ->>> jq {features, resource_id: .state["resources.jobs.foo"].__id__} .databricks/bundle/default/resources.json
        -{
        -  "features": {
        -    "deployment_history_shadow": {}
        -  },
        -  "resource_id": "[JOB_ID]"
        -}
        -
        -=== Unbind preserves the shadow history marker and leaves the job intact
        -
        ->>> [CLI] bundle deployment unbind foo
        -
        ->>> jq {features, state} .databricks/bundle/default/resources.json
        -{
        -  "features": {
        -    "deployment_history_shadow": {}
        -  },
        -  "state": {}
        -}
        -
        ->>> [CLI] jobs get [JOB_ID]
        -{
        -  "job_id": [JOB_ID]
        -}
        -
        -=== Rebind preserves the existing shadow history marker
        -
        ->>> [CLI] bundle deployment bind foo [JOB_ID] --auto-approve
        -Successfully bound job with an id '[JOB_ID]'
        -Run 'bundle deploy' to deploy changes to your workspace
        -
        ->>> jq {features, resource_id: .state["resources.jobs.foo"].__id__} .databricks/bundle/default/resources.json
        -{
        -  "features": {
        -    "deployment_history_shadow": {}
        -  },
        -  "resource_id": "[JOB_ID]"
        -}
        -
        ->>> print_requests.py --dms //versions --oneline
        -{"method": "POST", "path": "/api/2.0/bundle/deployments/[DEPLOYMENT_ID]/versions", "q": {"version_id": "7"}, "body": {"cli_version": "[CLI_VERSION]", "operations": [{"action_type": "OPERATION_ACTION_TYPE_UPDATE", "resource_key": "jobs.foo"}], "previous_version_id": "4", "version_type": "VERSION_TYPE_DEPLOY"}}
        -{"method": "PATCH", "path": "/api/2.0/bundle/deployments/[DEPLOYMENT_ID]/versions/7/operations/jobs.foo", "q": {"update_mask": "state,error_message,resource_id,status"}, "body": {"error_message": "", "resource_id": "[JOB_ID]", "sequence_id": 0, "state": "{\"state\":{\"deployment\":{\"kind\":\"BUNDLE\",\"metadata_file_path\":\"/Workspace/Users/[USERNAME]/.bundle/dms-bind-unbind-[UNIQUE_NAME]/default/state/metadata.json\"},\"edit_mode\":\"UI_LOCKED\",\"format\":\"MULTI_TASK\",\"max_concurrent_runs\":1,\"name\":\"foo\",\"queue\":{\"enabled\":true}}}", "status": "OPERATION_STATUS_SUCCEEDED"}}
        -{"method": "POST", "path": "/api/2.0/bundle/deployments/[DEPLOYMENT_ID]/versions/7/complete", "body": {"completion_reason": "VERSION_COMPLETE_SUCCESS"}}
        -
        -=== A subsequent deploy continues managing the same job
        -
        ->>> [CLI] bundle deploy
        -Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-bind-unbind-[UNIQUE_NAME]/default/files...
        -Files: 0 uploaded, 0 deleted
        -Resources: 0 created, 0 changed, 0 deleted, 1 unchanged
        -
        ->>> jq {features, resource_id: .state["resources.jobs.foo"].__id__} .databricks/bundle/default/resources.json
        -{
        -  "features": {
        -    "deployment_history_shadow": {}
        -  },
        -  "resource_id": "[JOB_ID]"
        -}
        -
        -=== A rejected duplicate bind creates no history version
        -
        ->>> musterr [CLI] bundle deployment bind foo [JOB_ID] --auto-approve
        -Error: Resource already managed
        -
        -The bundle is already managing resource for resources.jobs.foo and it is bound to the requested ID [JOB_ID].
        -
        -
        ->>> print_requests.py --dms //versions
        +Exit code: 1
        
    acceptance_test.go:1152: 
        LOG.cleanup: The following resources will be deleted:
        LOG.cleanup:   delete resources.jobs.foo
        LOG.cleanup: 
        LOG.cleanup: All files and directories at the following location will be deleted: /Workspace/Users/tester@databricks.com/.bundle/dms-bind-unbind-p65gzjceqbfylnstc5fsn4brke/default
        LOG.cleanup: 
        LOG.cleanup: Destroy: 1 deleted
        LOG.cleanup: Error: Not Found
        LOG.cleanup: 
        LOG.cleanup: Exit code: 1

trace $CLI bundle deployment bind foo $NEW_PIPELINE_ID --auto-approve &> out.bind-success.txt
trace print_requests.py '^//import-file/' '^//workspace/'

trace $CLI bundle summary -o json | jq .resources > out.summary.json

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added because now the summary contains deployment and version ID

print_requests.py '^//import-file/' '^//workspace/delete'

trace $CLI bundle summary -o json > out.summary.json
trace $CLI bundle summary -o json | nostamp.py > out.summary.json

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

updated nostamp.py to scrub the deployment ID from the summary output as well.

@shreyas-goenka
shreyas-goenka marked this pull request as ready for review November 10, 2026 13:46
@shreyas-goenka
shreyas-goenka requested review from a team as code owners November 10, 2026 13:46

@andrewnester andrewnester left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Deploy decides recording from config alone and deletes the shadow marker when experimental.deployment_history is false (process.go:376-384); destroy follows the marker alone; bind records when either the marker or the config says so. On a state that carries the marker, deployment_history: false plus bundle deploy records nothing and drops the marker, while bundle deployment bind on the same state still publishes a version. Is bind meant to follow the marker when the config turns recording off, or should it follow the config like deploy?

Nits:

  • bundle/phases/bind.go:292: the UpdateOperation error is returned unwrapped, so the warning reads "Failed to record bind history: Fault injected by test." with nothing saying which call failed; OperationBuffer.run wraps the same error as "recording operation for %s: %w".
  • libs/dms/metadata.go:156-159: the new first sentence and the retained "Lookup and return the deployment object from ..." say the same thing, and the TODO still proposes a dms.ReadDeployment now that this is dms.LookupDeployment.
  • .nextchanges/bundles/record-bind-history.md: "when binding ... when deployment history is enabled" stacks two "when" clauses; e.g. "Record a deployment history version for bundle deployment bind when deployment history is enabled."

Comment thread bundle/direct/bind.go
configured = configRoot.Experimental.DeploymentHistory
}
var dmsMetadata *DMSBindMetadata
if b.StateDB.RecordsHistory() || bundleenv.RecordsDeploymentHistory(ctx, configured) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This decides to record the bind in DMS, but nothing writes the deployment_history_shadow marker into the state that gets committed: UpgradeToWrite sees no marker and picks WritingFile, whose WAL header clears features (dstate/state.go:964), so after a first bind with history on, resources.json has features: null while DMS already lists jobs.foo under a new deployment. Every non-deploy command follows the marker (process.go:386, destroy.go:159), so bind followed by destroy records no destroy version and never deletes the deployment, and the node under the state path goes away with the state dir, leaving a DMS deployment that still lists the bound resource. Can bind set the marker when it records, the way process.go:376 does before deploy upgrades to shadow mode, and can bind-unbind grow a bind-then-destroy case to pin that down?

Comment thread bundle/phases/bind.go Outdated
}

// createBindingVersion claims a history version for the staged binding operations.
func createBindingVersion(ctx context.Context, b *bundle.Bundle, serial int, staged []bundledeployments.StagedOperation) (string, error) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is startVersion (dms.go:173) with previous/deploymentID taken from a lookup instead of the StateDB; likewise completeBindingHistory repeats DeploymentState.CompleteVersion, recordBindingOperation rebuilds what OperationUpdate.operation() and OperationBuffer.write already do (and hardcodes the 5s that historyDrainTimeoutPerOperation names), and createBindingDeployment repeats the create branch of createOrUpdateDeployment. The version-number comment in startVersion about why claiming the number up front is safe now lives in only one of the two copies. Can the request building move into shared helpers (a version-create taking deploymentID/previous/serial, a dms.CompleteVersion(ctx, w, name, success), an exported operation()), so bind and deploy cannot drift?

Comment thread bundle/phases/bind.go Outdated
defer cancel()
var state json.RawMessage
var resourceID string
if entry != nil {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

recordBindHistory always passes &metadata.ResourceState, so the entry == nil branch and the "or its removal" in the doc comment have no caller in this PR. Can this land with the unbind recording in #7401 that needs it, so the function here says only what it does?

Comment thread acceptance/bin/nostamp.py
if isinstance(data, dict):
for k in _PLAN_HEADER_KEYS:
data.pop(k, None) # 4. plan header, present only at the root
# 5. Strip the summary's DMS IDs, keeping deployment settings such as locks.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Running the DMS=true variants of the three scripts this PR pipes through nostamp (bind/pipelines/update, bind/pipelines/recreate, bind/dashboard/recreation) with the scripts at their base revision passes locally at this HEAD: after a first bind there is no marker, so summary and the state dump never carry bundle.deployment IDs. What difference is shape 5 and these three pipes guarding against, a cloud run or a later PR in the stack? If the latter, can they move there?

for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
got, err := parseLastVersionID(tt.deployment)
got, err := dms.ParseLastVersionID(tt.deployment)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This now tests dms.ParseLastVersionID from a different package. Can the table move to libs/dms/metadata_test.go next to the function, since the one in this file no longer exists?

Comment thread bundle/phases/bind.go
ResourceId string
}

// createBindingDeployment registers a first binding without changing metadata from an existing deploy.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"without changing metadata from an existing deploy" describes a case this function never sees: it only runs when current == nil (line 229), so there is no existing deploy to preserve. Something like "creates the deployment record when a bind is the first recorded operation; unlike deploy's createOrUpdateDeployment it reports no file_path because bind uploads no files, and the next deploy fills it in through StaleFields" would tell the reader what is actually different.

@shreyas-goenka
shreyas-goenka force-pushed the shreyas-goenka_data/shreyas-goenka/record-bind-history branch from 445c82f to 7b569b3 Compare November 13, 2026 12:58

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

DABs DABs related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants