Skip to content

direct: treat pipeline data_staging_options as a backend default - #7603

Draft
andrewnester wants to merge 2 commits into
mainfrom
factory/7580-pipeline-data-staging-options
Draft

andrewnester wants to merge 2 commits into
mainfrom
factory/7580-pipeline-data-staging-options

Conversation

@andrewnester

Copy link
Copy Markdown
Contributor

Changes

Fixes #7580. With the direct engine, a Lakeflow Connect pipeline that omits ingestion_definition.data_staging_options no longer shows a permanent update in bundle plan or gets updated (and restarted) on every deploy.

  • Add a backend_defaults rule for ingestion_definition.data_staging_options in pipelines.yml.

Why

The backend fills the field with the destination catalog/schema and a volume named after the pipeline id, so no config can ever match it. An explicitly configured data_staging_options still participates in drift detection because the rule only fires when the field is absent from both state and config.

Tests

New acceptance test acceptance/bundle/resources/pipelines/drift/data_staging_options (local only) covers the implicit no-op plan and a reported remote change on an explicit value.

This PR was written by Claude Code.
This pull request and its description were written by Isaac.

A Lakeflow Connect pipeline that omits ingestion_definition.data_staging_options
was planned as "update" on every `bundle plan` and updated on every deploy.
The backend fills the field with the destination catalog/schema and a volume
named __databricks_ingestion_gateway_staging_data-<pipeline-id>. The direct
engine saw a remote-only value with no matching rule and reported it as drift,
although no config can ever match it because it embeds the pipeline's own id.
Continuous pipelines were restarted by every deploy because of this.

Add a `backend_defaults` rule without pinned values for
ingestion_definition.data_staging_options in pipelines.yml. It only fires when
the field is absent from both state and config, so an explicitly configured
data_staging_options still participates in drift detection.

Teach the fake server to fill in data_staging_options like the backend when
ingestion_definition is set, the field is omitted, and a destination catalog
exists, and add an acceptance test covering both the implicit (no-op plan) and
explicit (remote change reported) cases.

Fixes #7580

Co-authored-by: Isaac <no-reply@databricks.com>
@andrewnester andrewnester added the ai-auto PRs made by AI loops & factories, label Nov 13, 2026
@github-actions github-actions Bot added the DABs DABs related issues label Nov 13, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Commit: 9d316654c | Workflow run

Regression test report

Selected 1 changed tests (limit=20, 0 not selected)

2 tests: regression=2 unreleased=0 coverage=0 failing=0
test branch main (8e70c8b) latest (v1.26.0)
TestAccept/bundle/resources/pipelines/drift/data_staging_options/DMS= ✅ ❌ ❌
TestAccept/bundle/resources/pipelines/drift/data_staging_options/DMS=true ✅ ❌ ❌
TestAccept/bundle/resources/pipelines/drift/data_staging_options/DMS=: branch ✅ | main (8e70c8b) ❌ | latest (v1.26.0) ❌

main (8e70c8b):

acceptance_test.go:1374: Diff:
        --- bundle/resources/pipelines/drift/data_staging_options/output.txt
        +++ /tmp/TestAcceptbundleresourcespipelinesdriftdata_staging_options3936172368/002/output.txt
        @@ -11,8 +11,10 @@
         === Plan is a no-op: the backend-filled data_staging_options is a default, not drift
         
         >>> [CLI] bundle plan
        -Plan: 0 to add, 0 to change, 0 to delete, 2 unchanged
        +update pipelines.implicit_staging
         
        +Plan: 0 to add, 1 to change, 0 to delete, 1 unchanged
        +
         >>> [CLI] bundle plan -o json
         {
           "resources.pipelines.explicit_staging": {
        @@ -20,11 +21,10 @@
             "changes": null
           },
           "resources.pipelines.implicit_staging": {
        -    "action": "skip",
        +    "action": "update",
             "changes": {
               "ingestion_definition.data_staging_options": {
        -        "action": "skip",
        -        "reason": "backend_default",
        +        "action": "update",
                 "remote": {
                   "catalog_name": "my_catalog",
                   "schema_name": "my_schema",
        @@ -39,46 +37,14 @@
         
         >>> [CLI] bundle deploy
         Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/pipeline-data-staging-options/default/files...
        +Updated pipelines.implicit_staging
         Files: 0 uploaded, 0 deleted
        -Resources: 0 created, 0 changed, 0 deleted, 2 unchanged
        +Resources: 0 created, 1 changed, 0 deleted, 1 unchanged
         
         >>> print_requests.py --sort //api/2.0/pipelines
        +contains error: '"method": "PUT"' was not expected: '"method": "PUT",'
         json[1].body.ingestion_definition.data_staging_options.catalog_name = "staging_catalog";
         json[1].body.ingestion_definition.data_staging_options.schema_name = "staging_schema";
         json[1].body.ingestion_definition.data_staging_options.volume_name = "staging_volume";
         
        -=== Explicitly configured data_staging_options still detects a remote change
        -
        ->>> [CLI] bundle plan
        -update pipelines.explicit_staging
        -
        -Plan: 0 to add, 1 to change, 0 to delete, 1 unchanged
        -
        ->>> [CLI] bundle plan -o json
        -{
        -  "resources.pipelines.explicit_staging": {
        -    "action": "update",
        -    "changes": {
        -      "ingestion_definition.data_staging_options.volume_name": {
        -        "action": "update",
        -        "old": "staging_volume",
        -        "new": "staging_volume",
        -        "remote": "changed_volume"
        -      }
        -    }
        -  },
        -  "resources.pipelines.implicit_staging": {
        -    "action": "skip",
        -    "changes": {
        -      "ingestion_definition.data_staging_options": {
        -        "action": "skip",
        -        "reason": "backend_default",
        -        "remote": {
        -          "catalog_name": "my_catalog",
        -          "schema_name": "my_schema",
        -          "volume_name": "__databricks_ingestion_gateway_staging_data-[UUID]"
        -        }
        -      }
        -    }
        -  }
        -}
        +Exit code: 1
        
    acceptance_test.go:1164: Test produced unexpected files:
        .databricks/.gitignore
        .databricks/bundle/default/deployment.json
        .databricks/bundle/default/resources.json
        .databricks/bundle/default/sync-snapshots/1a4f00fb9989dcdb.json
FAIL

latest (v1.26.0):

acceptance_test.go:1374: Diff:
        --- bundle/resources/pipelines/drift/data_staging_options/output.txt
        +++ /tmp/TestAcceptbundleresourcespipelinesdriftdata_staging_options1250739129/002/output.txt
        @@ -11,8 +11,10 @@
         === Plan is a no-op: the backend-filled data_staging_options is a default, not drift
         
         >>> [CLI] bundle plan
        -Plan: 0 to add, 0 to change, 0 to delete, 2 unchanged
        +update pipelines.implicit_staging
         
        +Plan: 0 to add, 1 to change, 0 to delete, 1 unchanged
        +
         >>> [CLI] bundle plan -o json
         {
           "resources.pipelines.explicit_staging": {
        @@ -20,11 +21,10 @@
             "changes": null
           },
           "resources.pipelines.implicit_staging": {
        -    "action": "skip",
        +    "action": "update",
             "changes": {
               "ingestion_definition.data_staging_options": {
        -        "action": "skip",
        -        "reason": "backend_default",
        +        "action": "update",
                 "remote": {
                   "catalog_name": "my_catalog",
                   "schema_name": "my_schema",
        @@ -39,46 +37,14 @@
         
         >>> [CLI] bundle deploy
         Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/pipeline-data-staging-options/default/files...
        +Updated pipelines.implicit_staging
         Files: 0 uploaded, 0 deleted
        -Resources: 0 created, 0 changed, 0 deleted, 2 unchanged
        +Resources: 0 created, 1 changed, 0 deleted, 1 unchanged
         
         >>> print_requests.py --sort //api/2.0/pipelines
        +contains error: '"method": "PUT"' was not expected: '"method": "PUT",'
         json[1].body.ingestion_definition.data_staging_options.catalog_name = "staging_catalog";
         json[1].body.ingestion_definition.data_staging_options.schema_name = "staging_schema";
         json[1].body.ingestion_definition.data_staging_options.volume_name = "staging_volume";
         
        -=== Explicitly configured data_staging_options still detects a remote change
        -
        ->>> [CLI] bundle plan
        -update pipelines.explicit_staging
        -
        -Plan: 0 to add, 1 to change, 0 to delete, 1 unchanged
        -
        ->>> [CLI] bundle plan -o json
        -{
        -  "resources.pipelines.explicit_staging": {
        -    "action": "update",
        -    "changes": {
        -      "ingestion_definition.data_staging_options.volume_name": {
        -        "action": "update",
        -        "old": "staging_volume",
        -        "new": "staging_volume",
        -        "remote": "changed_volume"
        -      }
        -    }
        -  },
        -  "resources.pipelines.implicit_staging": {
        -    "action": "skip",
        -    "changes": {
        -      "ingestion_definition.data_staging_options": {
        -        "action": "skip",
        -        "reason": "backend_default",
        -        "remote": {
        -          "catalog_name": "my_catalog",
        -          "schema_name": "my_schema",
        -          "volume_name": "__databricks_ingestion_gateway_staging_data-[UUID]"
        -        }
        -      }
        -    }
        -  }
        -}
        +Exit code: 1
        
    acceptance_test.go:1164: Test produced unexpected files:
        .databricks/.gitignore
        .databricks/bundle/default/deployment.json
        .databricks/bundle/default/resources.json
        .databricks/bundle/default/sync-snapshots/489729abf9d685b5.json
FAIL
TestAccept/bundle/resources/pipelines/drift/data_staging_options/DMS=true: branch ✅ | main (8e70c8b) ❌ | latest (v1.26.0) ❌

main (8e70c8b):

acceptance_test.go:1374: Diff:
        --- bundle/resources/pipelines/drift/data_staging_options/output.txt
        +++ /tmp/TestAcceptbundleresourcespipelinesdriftdata_staging_options1721353783/002/output.txt
        @@ -11,8 +11,10 @@
         === Plan is a no-op: the backend-filled data_staging_options is a default, not drift
         
         >>> [CLI] bundle plan
        -Plan: 0 to add, 0 to change, 0 to delete, 2 unchanged
        +update pipelines.implicit_staging
         
        +Plan: 0 to add, 1 to change, 0 to delete, 1 unchanged
        +
         >>> [CLI] bundle plan -o json
         {
           "resources.pipelines.explicit_staging": {
        @@ -20,11 +21,10 @@
             "changes": null
           },
           "resources.pipelines.implicit_staging": {
        -    "action": "skip",
        +    "action": "update",
             "changes": {
               "ingestion_definition.data_staging_options": {
        -        "action": "skip",
        -        "reason": "backend_default",
        +        "action": "update",
                 "remote": {
                   "catalog_name": "my_catalog",
                   "schema_name": "my_schema",
        @@ -39,46 +37,14 @@
         
         >>> [CLI] bundle deploy
         Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/pipeline-data-staging-options/default/files...
        +Updated pipelines.implicit_staging
         Files: 0 uploaded, 0 deleted
        -Resources: 0 created, 0 changed, 0 deleted, 2 unchanged
        +Resources: 0 created, 1 changed, 0 deleted, 1 unchanged
         
         >>> print_requests.py --sort //api/2.0/pipelines
        +contains error: '"method": "PUT"' was not expected: '"method": "PUT",'
         json[1].body.ingestion_definition.data_staging_options.catalog_name = "staging_catalog";
         json[1].body.ingestion_definition.data_staging_options.schema_name = "staging_schema";
         json[1].body.ingestion_definition.data_staging_options.volume_name = "staging_volume";
         
        -=== Explicitly configured data_staging_options still detects a remote change
        -
        ->>> [CLI] bundle plan
        -update pipelines.explicit_staging
        -
        -Plan: 0 to add, 1 to change, 0 to delete, 1 unchanged
        -
        ->>> [CLI] bundle plan -o json
        -{
        -  "resources.pipelines.explicit_staging": {
        -    "action": "update",
        -    "changes": {
        -      "ingestion_definition.data_staging_options.volume_name": {
        -        "action": "update",
        -        "old": "staging_volume",
        -        "new": "staging_volume",
        -        "remote": "changed_volume"
        -      }
        -    }
        -  },
        -  "resources.pipelines.implicit_staging": {
        -    "action": "skip",
        -    "changes": {
        -      "ingestion_definition.data_staging_options": {
        -        "action": "skip",
        -        "reason": "backend_default",
        -        "remote": {
        -          "catalog_name": "my_catalog",
        -          "schema_name": "my_schema",
        -          "volume_name": "__databricks_ingestion_gateway_staging_data-[UUID]"
        -        }
        -      }
        -    }
        -  }
        -}
        +Exit code: 1
        
    acceptance_test.go:1164: Test produced unexpected files:
        .databricks/.gitignore
        .databricks/bundle/default/deployment.json
        .databricks/bundle/default/resources.json
        .databricks/bundle/default/sync-snapshots/621b8140f890aa37.json

latest (v1.26.0):

acceptance_test.go:1374: Diff:
        --- bundle/resources/pipelines/drift/data_staging_options/output.txt
        +++ /tmp/TestAcceptbundleresourcespipelinesdriftdata_staging_options2569988936/002/output.txt
        @@ -11,8 +11,10 @@
         === Plan is a no-op: the backend-filled data_staging_options is a default, not drift
         
         >>> [CLI] bundle plan
        -Plan: 0 to add, 0 to change, 0 to delete, 2 unchanged
        +update pipelines.implicit_staging
         
        +Plan: 0 to add, 1 to change, 0 to delete, 1 unchanged
        +
         >>> [CLI] bundle plan -o json
         {
           "resources.pipelines.explicit_staging": {
        @@ -20,11 +21,10 @@
             "changes": null
           },
           "resources.pipelines.implicit_staging": {
        -    "action": "skip",
        +    "action": "update",
             "changes": {
               "ingestion_definition.data_staging_options": {
        -        "action": "skip",
        -        "reason": "backend_default",
        +        "action": "update",
                 "remote": {
                   "catalog_name": "my_catalog",
                   "schema_name": "my_schema",
        @@ -39,46 +37,14 @@
         
         >>> [CLI] bundle deploy
         Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/pipeline-data-staging-options/default/files...
        +Updated pipelines.implicit_staging
         Files: 0 uploaded, 0 deleted
        -Resources: 0 created, 0 changed, 0 deleted, 2 unchanged
        +Resources: 0 created, 1 changed, 0 deleted, 1 unchanged
         
         >>> print_requests.py --sort //api/2.0/pipelines
        +contains error: '"method": "PUT"' was not expected: '"method": "PUT",'
         json[1].body.ingestion_definition.data_staging_options.catalog_name = "staging_catalog";
         json[1].body.ingestion_definition.data_staging_options.schema_name = "staging_schema";
         json[1].body.ingestion_definition.data_staging_options.volume_name = "staging_volume";
         
        -=== Explicitly configured data_staging_options still detects a remote change
        -
        ->>> [CLI] bundle plan
        -update pipelines.explicit_staging
        -
        -Plan: 0 to add, 1 to change, 0 to delete, 1 unchanged
        -
        ->>> [CLI] bundle plan -o json
        -{
        -  "resources.pipelines.explicit_staging": {
        -    "action": "update",
        -    "changes": {
        -      "ingestion_definition.data_staging_options.volume_name": {
        -        "action": "update",
        -        "old": "staging_volume",
        -        "new": "staging_volume",
        -        "remote": "changed_volume"
        -      }
        -    }
        -  },
        -  "resources.pipelines.implicit_staging": {
        -    "action": "skip",
        -    "changes": {
        -      "ingestion_definition.data_staging_options": {
        -        "action": "skip",
        -        "reason": "backend_default",
        -        "remote": {
        -          "catalog_name": "my_catalog",
        -          "schema_name": "my_schema",
        -          "volume_name": "__databricks_ingestion_gateway_staging_data-[UUID]"
        -        }
        -      }
        -    }
        -  }
        -}
        +Exit code: 1
        
    acceptance_test.go:1164: Test produced unexpected files:
        .databricks/.gitignore
        .databricks/bundle/default/deployment.json
        .databricks/bundle/default/resources.json
        .databricks/bundle/default/sync-snapshots/1a5da8f17e22a858.json

@eng-dev-ecosystem-bot

Copy link
Copy Markdown
Collaborator

✅ Integration test report

Commit: 9d31665

Run: 43444701810

Env 💚​RECOVERED ✅​pass 🙈​skip Time
💚​ aws linux-2core-8gb 1 270 16 10:42
💚​ aws-windows-latest-4core-16gb 1 272 14 6:35
💚​ azure linux-2core-8gb 1 269 16 8:50
💚​ azure-windows-latest-4core-16gb 1 271 14 6:12
💚​ gcp linux-2core-8gb 1 270 16 7:47
💚​ gcp-windows-latest-4core-16gb 1 272 14 5:14
Test Name aws linux-2core-8gb aws-windows-latest-4core-16gb azure linux-2core-8gb azure-windows-latest-4core-16gb gcp linux-2core-8gb gcp-windows-latest-4core-16gb
💚​ TestAccept 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
Top 6 slowest tests (at least 2 minutes):
duration env testname
3:21 gcp-windows-latest-4core-16gb TestAccept
3:15 aws-windows-latest-4core-16gb TestAccept
3:04 azure-windows-latest-4core-16gb TestAccept
2:22 aws linux-2core-8gb TestFilerWorkspaceFilesExtensionsReadDir
2:17 aws-windows-latest-4core-16gb TestSyncFullFileSync
2:10 azure linux-2core-8gb TestImportDir

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ai-auto PRs made by AI loops & factories, DABs DABs related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Direct engine: Lakeflow Connect CDC pipeline shows a permanent diff on ingestion_definition.data_staging_options

2 participants