You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Direct engine: Lakeflow Connect CDC pipeline shows a permanent diff on ingestion_definition.data_staging_options #7580
With the direct deployment engine, a Lakeflow Connect CDC ingestion pipeline that does not set ingestion_definition.data_staging_options shows an update on every bundle plan, even with no config changes. Deploying it updates the pipeline. For a continuous pipeline, this means a restart on every deploy.
The backend fills in the default staging location (the destination catalog and schema, plus a volume named after the pipeline ID). The CLI then sees a value that is in remote but not in config, and plans an update. The value cannot be set to match in config, because volume_name contains the pipeline's own ID.
bundle/direct/dresources/configs/pipelines.yml has no backend_defaults entry for this field, on v1.23.0 or main.
And running continuous pipelines are then updated and interrupted
OS and CLI version
Databricks CLI v1.23.0, Linux (GitHub Actions). Also present on main.
Is this a regression?
Not known.
Detailed plan
See Actual Behavior above.
Debug Logs
Not included.
Suggested fix
Add a backend_defaults entry to bundle/direct/dresources/configs/pipelines.yml, similar to #7577:
backend_defaults:
# Lakeflow Connect fills in the default staging location (destination catalog/schema and a# __databricks_ingestion_gateway_staging_data-<pipeline-id> volume) when it is not set.
- field: ingestion_definition.data_staging_options
An explicitly configured data_staging_options would still be compared as usual.
Activity