Skip to content

Direct engine: Lakeflow Connect CDC pipeline shows a permanent diff on ingestion_definition.data_staging_options #7580

Open

Description

@troyel

Describe the issue

With the direct deployment engine, a Lakeflow Connect CDC ingestion pipeline that does not set ingestion_definition.data_staging_options shows an update on every bundle plan, even with no config changes. Deploying it updates the pipeline. For a continuous pipeline, this means a restart on every deploy.

The backend fills in the default staging location (the destination catalog and schema, plus a volume named after the pipeline ID). The CLI then sees a value that is in remote but not in config, and plans an update. The value cannot be set to match in config, because volume_name contains the pipeline's own ID.

bundle/direct/dresources/configs/pipelines.yml has no backend_defaults entry for this field, on v1.23.0 or main.

Configuration

resources:
  pipelines:
    cdc_ingestion:
      name: cdc_ingestion
      catalog: my_catalog
      schema: my_schema
      continuous: true
      ingestion_definition:
        connection_name: my_sqlserver_connection
        connector_type: CDC
        objects:
          - table:
              source_catalog: src_db
              source_schema: dbo
              source_table: my_table
              destination_catalog: my_catalog
              destination_schema: my_schema

Steps to reproduce the behavior

  1. Run databricks bundle deploy -t <target> with the pipeline above.
  2. Run databricks bundle plan -t <target> -o json without changing anything.
  3. The pipeline is planned as update. Each later deploy updates it again.

Expected Behavior

No change is planned, because the remote value is a backend default for a field the config does not set.

Actual Behavior

"resources.pipelines.cdc_ingestion": {
  "action": "update",
  "changes": {
    "ingestion_definition.data_staging_options": {
      "action": "update",
      "remote": {
        "catalog_name": "my_catalog",
        "schema_name": "my_schema",
        "volume_name": "__databricks_ingestion_gateway_staging_data-<pipeline-id>"
      }
    }
  }
}

And running continuous pipelines are then updated and interrupted

OS and CLI version

Databricks CLI v1.23.0, Linux (GitHub Actions). Also present on main.

Is this a regression?

Not known.

Detailed plan

See Actual Behavior above.

Debug Logs

Not included.

Suggested fix

Add a backend_defaults entry to bundle/direct/dresources/configs/pipelines.yml, similar to #7577:

backend_defaults:
  # Lakeflow Connect fills in the default staging location (destination catalog/schema and a
  # __databricks_ingestion_gateway_staging_data-<pipeline-id> volume) when it is not set.
  - field: ingestion_definition.data_staging_options

An explicitly configured data_staging_options would still be compared as usual.

Activity

  1. self-assigned this
    on Nov 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

BugSomething isn't workingDABsDABs related issues

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions