Skip to content

[draft] Apply configured MLflow experiment tags on bundle update - #7302

Closed
lennartkats-db wants to merge 4 commits into
mainfrom
factory/dabs/3013c1dca491
Closed

lennartkats-db wants to merge 4 commits into
mainfrom
factory/dabs/3013c1dca491

Conversation

@lennartkats-db

@lennartkats-db lennartkats-db commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Changes

  • Apply newly configured and changed MLflow experiment tags on bundle redeploy without recreating the experiment.
  • Compare tags by key and reject removal of a previously managed tag when it remains on the experiment. The MLflow experiment API does not provide tag deletion, so this avoids reporting a successful deploy that leaves the tag active. Tags added by the backend remain unmanaged.

Why

An existing experiment ignored new or changed tags in the bundle configuration, including mlflow.workload_creation_policy_id. Recreating an experiment to apply them would discard its runs or traces. Fixes #6521.

Tests

  • go test ./bundle/direct/dresources -count=1
  • go test ./acceptance -run 'TestAccept/bundle/resources/experiments/basic' -tail -test.v -count=1 (both DMS modes)
  • ./task check-changelog

This PR was written by Codex.

@lennartkats-db lennartkats-db added DABs DABs related issues software-factory PR produced by the software factories task labels Nov 3, 2026
@github-actions

github-actions Bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Commit: 4304677d0 | Workflow run

Regression test report

Selected 1 changed tests (limit=20, 0 not selected)

2 tests: regression=2 unreleased=0 coverage=0 failing=0
test branch main (cf85b49) latest (v1.26.0)
TestAccept/bundle/resources/experiments/basic/DMS= ✅ ❌ ❌
TestAccept/bundle/resources/experiments/basic/DMS=true ✅ ❌ ❌
TestAccept/bundle/resources/experiments/basic/DMS=: branch ✅ | main (cf85b49) ❌ | latest (v1.26.0) ❌

main (cf85b49):

acceptance_test.go:1374: Diff:
        --- bundle/resources/experiments/basic/output.txt
        +++ /tmp/TestAcceptbundleresourcesexperimentsbasicDMS=2860291882/002/output.txt
        @@ -154,10 +154,6 @@
               "value": "v1"
             },
             {
        -      "key": "k2",
        -      "value": "v2"
        -    },
        -    {
               "key": "mlflow.experiment.sourceName",
               "value": "/Users/[USERNAME]/new-name-[UNIQUE_NAME]"
             },
        @@ -185,53 +181,9 @@
         >>> envsubst
         
         >>> musterr [CLI] bundle deploy -qq
        -Error: cannot update resources.experiments.my_experiment: updating id=[MY_EXPERIMENT_ID_3]: cannot remove experiment tag "k1": the MLflow API does not support deleting experiment tags
         
        +Unexpected success
         
        -=== removing a configured tag is rejected
        -
        ->>> envsubst
        -
        ->>> musterr [CLI] bundle deploy -qq
        -Error: cannot update resources.experiments.my_experiment: updating id=[MY_EXPERIMENT_ID_3]: cannot remove experiment tag "k2": the MLflow API does not support deleting experiment tags
        -
        -
        ->>> [CLI] experiments get-experiment [MY_EXPERIMENT_ID_3]
        -{
        -  "name": "/Users/[USERNAME]/new-name-[UNIQUE_NAME]",
        -  "artifact_location": "s3://new-[UNIQUE_NAME]",
        -  "tags": [
        -    {
        -      "key": "k1",
        -      "value": "v1"
        -    },
        -    {
        -      "key": "k2",
        -      "value": "v2"
        -    },
        -    {
        -      "key": "mlflow.experiment.sourceName",
        -      "value": "/Users/[USERNAME]/new-name-[UNIQUE_NAME]"
        -    },
        -    {
        -      "key": "mlflow.experimentType",
        -      "value": "MLFLOW_EXPERIMENT"
        -    },
        -    {
        -      "key": "mlflow.ownerEmail",
        -      "value": "[USERNAME]"
        -    },
        -    {
        -      "key": "mlflow.ownerId",
        -      "value": "[USERID]"
        -    },
        -    {
        -      "key": "mlflow.ownerId",
        -      "value": "[USERID]"
        -    }
        -  ]
        -}
        -
         >>> [CLI] bundle destroy --auto-approve
         The following resources will be deleted:
           delete resources.experiments.my_experiment
        @@ -239,3 +189,5 @@
         All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/experiment-basic-[UNIQUE_NAME]/default
         
         Destroy: 1 deleted
        +
        +Exit code: 1
        
    acceptance_test.go:1374: 
        	Error Trace:	/home/runner/work/cli/cli/libs/testdiff/testdiff.go:22
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1374
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1127
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:678
        	Error:      	Not equal: 
        	            	expected: "update experiments.my_experiment\n\nPlan: 0 to add, 1 to change, 0 to delete, 0 unchanged\n"
        	            	actual  : "Plan: 0 to add, 0 to change, 0 to delete, 1 unchanged\n"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -1,4 +1,2 @@
        	            	-update experiments.my_experiment
        	            	+Plan: 0 to add, 0 to change, 0 to delete, 1 unchanged
        	            	 
        	            	-Plan: 0 to add, 1 to change, 0 to delete, 0 unchanged
        	            	-
        	Test:       	TestAccept/bundle/resources/experiments/basic/DMS=
        	Messages:   	bundle/resources/experiments/basic/out.plan.txt vs /tmp/TestAcceptbundleresourcesexperimentsbasicDMS=2860291882/001/out.plan.txt
    acceptance_test.go:1164: Test produced unexpected files:
        .databricks/.gitignore
        .databricks/bundle/default/deployment.json

latest (v1.26.0): same output as main (cf85b49)

TestAccept/bundle/resources/experiments/basic/DMS=true: branch ✅ | main (cf85b49) ❌ | latest (v1.26.0) ❌

main (cf85b49):

acceptance_test.go:1374: Diff:
        --- bundle/resources/experiments/basic/output.txt
        +++ /tmp/TestAcceptbundleresourcesexperimentsbasicDMS=true401510761/002/output.txt
        @@ -154,10 +154,6 @@
               "value": "v1"
             },
             {
        -      "key": "k2",
        -      "value": "v2"
        -    },
        -    {
               "key": "mlflow.experiment.sourceName",
               "value": "/Users/[USERNAME]/new-name-[UNIQUE_NAME]"
             },
        @@ -185,53 +181,9 @@
         >>> envsubst
         
         >>> musterr [CLI] bundle deploy -qq
        -Error: cannot update resources.experiments.my_experiment: updating id=[MY_EXPERIMENT_ID_3]: cannot remove experiment tag "k1": the MLflow API does not support deleting experiment tags
         
        +Unexpected success
         
        -=== removing a configured tag is rejected
        -
        ->>> envsubst
        -
        ->>> musterr [CLI] bundle deploy -qq
        -Error: cannot update resources.experiments.my_experiment: updating id=[MY_EXPERIMENT_ID_3]: cannot remove experiment tag "k2": the MLflow API does not support deleting experiment tags
        -
        -
        ->>> [CLI] experiments get-experiment [MY_EXPERIMENT_ID_3]
        -{
        -  "name": "/Users/[USERNAME]/new-name-[UNIQUE_NAME]",
        -  "artifact_location": "s3://new-[UNIQUE_NAME]",
        -  "tags": [
        -    {
        -      "key": "k1",
        -      "value": "v1"
        -    },
        -    {
        -      "key": "k2",
        -      "value": "v2"
        -    },
        -    {
        -      "key": "mlflow.experiment.sourceName",
        -      "value": "/Users/[USERNAME]/new-name-[UNIQUE_NAME]"
        -    },
        -    {
        -      "key": "mlflow.experimentType",
        -      "value": "MLFLOW_EXPERIMENT"
        -    },
        -    {
        -      "key": "mlflow.ownerEmail",
        -      "value": "[USERNAME]"
        -    },
        -    {
        -      "key": "mlflow.ownerId",
        -      "value": "[USERID]"
        -    },
        -    {
        -      "key": "mlflow.ownerId",
        -      "value": "[USERID]"
        -    }
        -  ]
        -}
        -
         >>> [CLI] bundle destroy --auto-approve
         The following resources will be deleted:
           delete resources.experiments.my_experiment
        @@ -239,3 +189,5 @@
         All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/experiment-basic-[UNIQUE_NAME]/default
         
         Destroy: 1 deleted
        +
        +Exit code: 1
        
    acceptance_test.go:1374: 
        	Error Trace:	/home/runner/work/cli/cli/libs/testdiff/testdiff.go:22
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1374
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1127
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:678
        	Error:      	Not equal: 
        	            	expected: "update experiments.my_experiment\n\nPlan: 0 to add, 1 to change, 0 to delete, 0 unchanged\n"
        	            	actual  : "Plan: 0 to add, 0 to change, 0 to delete, 1 unchanged\n"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -1,4 +1,2 @@
        	            	-update experiments.my_experiment
        	            	+Plan: 0 to add, 0 to change, 0 to delete, 1 unchanged
        	            	 
        	            	-Plan: 0 to add, 1 to change, 0 to delete, 0 unchanged
        	            	-
        	Test:       	TestAccept/bundle/resources/experiments/basic/DMS=true
        	Messages:   	bundle/resources/experiments/basic/out.plan.txt vs /tmp/TestAcceptbundleresourcesexperimentsbasicDMS=true401510761/001/out.plan.txt
    acceptance_test.go:1164: Test produced unexpected files:
        .databricks/.gitignore
        .databricks/bundle/default/deployment.json
FAIL

latest (v1.26.0): same output as main (cf85b49)

@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Collaborator

✅ Integration test report

Commit: 4304677

Run: 43412828475

Env 💚​RECOVERED ✅​pass 🙈​skip Time
💚​ aws linux-2core-8gb 1 273 15 9:39
💚​ aws-windows-latest-4core-16gb 1 275 13 8:22
💚​ azure linux-2core-8gb 1 272 15 8:54
💚​ azure-windows-latest-4core-16gb 1 274 13 6:56
💚​ gcp linux-2core-8gb 1 273 15 10:07
💚​ gcp-windows-latest-4core-16gb 1 275 13 7:07
Test Name aws linux-2core-8gb aws-windows-latest-4core-16gb azure linux-2core-8gb azure-windows-latest-4core-16gb gcp linux-2core-8gb gcp-windows-latest-4core-16gb
💚​ TestAccept 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
Top 17 slowest tests (at least 2 minutes):
duration env testname
4:41 aws-windows-latest-4core-16gb TestAccept/bundle/resources/experiments/basic/DMS=true
4:24 azure-windows-latest-4core-16gb TestAccept
3:55 gcp-windows-latest-4core-16gb TestAccept
3:52 gcp linux-2core-8gb TestAccept
3:41 azure linux-2core-8gb TestAccept/bundle/resources/experiments/basic/DMS=true
3:35 aws-windows-latest-4core-16gb TestAccept
3:26 aws linux-2core-8gb TestAccept
3:15 azure linux-2core-8gb TestAccept
3:06 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/experiments/basic/DMS=
2:47 aws linux-2core-8gb TestAccept/bundle/resources/experiments/basic/DMS=true
2:30 gcp linux-2core-8gb TestAccept/bundle/resources/experiments/basic/DMS=
2:28 gcp linux-2core-8gb TestAccept/bundle/resources/experiments/basic/DMS=true
2:27 aws-windows-latest-4core-16gb TestAccept/bundle/resources/experiments/basic/DMS=
2:27 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/experiments/basic/DMS=true
2:26 azure-windows-latest-4core-16gb TestAccept/bundle/resources/experiments/basic/DMS=
2:12 azure-windows-latest-4core-16gb TestAccept/bundle/resources/experiments/basic/DMS=true
2:01 gcp-windows-latest-4core-16gb TestFilerWorkspaceFilesExtensionsDelete

@lennartkats-db lennartkats-db added software-factory PR produced by the software factories task and removed software-factory PR produced by the software factories task labels Nov 3, 2026
@lennartkats-db
lennartkats-db marked this pull request as ready for review November 4, 2026 19:21
@lennartkats-db
lennartkats-db requested review from a team as code owners November 4, 2026 19:21
@lennartkats-db
lennartkats-db marked this pull request as draft November 4, 2026 19:21
@lennartkats-db
lennartkats-db marked this pull request as ready for review November 4, 2026 22:00
@lennartkats-db lennartkats-db changed the title [draft] Fix bundle/direct/dresources/experiment.go [draft] Apply configured MLflow experiment tags on bundle update Nov 5, 2026
@lennartkats-db lennartkats-db added the ai-auto PRs made by AI loops & factories, label Nov 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ai-auto PRs made by AI loops & factories, DABs DABs related issues software-factory PR produced by the software factories task

Projects

None yet

Development

Successfully merging this pull request may close these issues.

DAB does not update MLflow Experiment tags on redeploy

2 participants