Skip to content

Apply resource lifecycle metadata in config-remote-sync change detection - #7645

Open
ilyakuz-db wants to merge 4 commits into
mainfrom
ilyakuz-db/configsync-lifecycle-metadata
Open

ilyakuz-db wants to merge 4 commits into
mainfrom
ilyakuz-db/configsync-lifecycle-metadata

Conversation

@ilyakuz-db

Copy link
Copy Markdown
Contributor

Changes

  • config-remote-sync now applies the direct engine's lifecycle metadata to every change, whatever action the plan gave it (bundle/configsync/lifecycle.go). A remote value is never written back when it falls under ignore_remote_changes (output-only, input-only, backend-managed), or when it's a backend default the config doesn't set.
  • When the plan reports a whole object (one side is nil), those fields are left out of it, and the change is skipped if nothing else differs. Inside a remote-only object, only backend defaults pinned to allowed values are dropped. A field like node_type_id in a task added in the UI may be the user's pick, so it's kept.
  • Job run_as keys are still synced, so a principal kind switch keeps working ([DECO-28500] Fix config-remote-sync writing run_as: {} on a principal kind switch #7493).
  • Moved the backend-default matcher from bundle/direct to dresources.ResourceLifecycleConfig.IsBackendDefault so the plan and config sync share it. The plan's behavior doesn't change.

Why

Config sync only dropped changes the plan marked as skipped. Metadata-covered fields still reached the YAML whenever the plan reported an update, or reported the object that contains them:

  • A genie space's etag is output_only, but OverrideChangeDesc promotes etag drift to an update for deploy, so the etag ended up in the YAML.
  • Input-only fields are never returned by GET, so sync removed them when the config had an undeployed edit (a job's trigger.table_update.condition), or when the whole object was missing remotely (a serving endpoint's telemetry_config.table_names).
  • Output-only and backend-managed fields inside an object added in the UI were written as-is (for example aws_attributes under a for-each task's cluster).
  • Backend defaults were added back: a job's format once the config dropped it, or spark.sql.ansi.enabled inside a pipeline configuration map added remotely.

Tests

  • New unit test in diff_test.go covering each case above, plus the run_as switch and a backend default with a non-default value. It fails without the fix.
  • go test ./bundle/..., the config-remote-sync acceptance tests and golangci-lint pass locally.

This pull request and its description were written by Isaac.

ilyakuz-db and others added 4 commits November 15, 2026 23:47
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
@ilyakuz-db
ilyakuz-db requested review from a team as code owners November 16, 2026 00:08
@ilyakuz-db ilyakuz-db added DABs DABs related issues software-factory PR produced by the software factories task ai-auto PRs made by AI loops & factories, labels Nov 16, 2026
@ilyakuz-db
ilyakuz-db requested a review from denik November 16, 2026 00:08
@ilyakuz-db ilyakuz-db self-assigned this Nov 16, 2026
@eng-dev-ecosystem-bot

Copy link
Copy Markdown
Collaborator

✅ Integration test report

Commit: 2d69c06

Run: 43778640015

Env 💚​RECOVERED ✅​pass 🙈​skip Time
💚​ aws linux-2core-8gb 1 270 15 3:35
💚​ aws-windows-latest-4core-16gb 1 272 13 3:04
💚​ azure linux-2core-8gb 1 269 15 3:25
💚​ azure-windows-latest-4core-16gb 1 271 13 3:04
💚​ gcp linux-2core-8gb 1 270 15 3:47
💚​ gcp-windows-latest-4core-16gb 1 272 13 3:35
Test Name aws linux-2core-8gb aws-windows-latest-4core-16gb azure linux-2core-8gb azure-windows-latest-4core-16gb gcp linux-2core-8gb gcp-windows-latest-4core-16gb
💚​ TestAccept 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
Top 3 slowest tests (at least 2 minutes):
duration env testname
3:31 gcp-windows-latest-4core-16gb TestAccept
3:00 aws-windows-latest-4core-16gb TestAccept
3:00 azure-windows-latest-4core-16gb TestAccept

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ai-auto PRs made by AI loops & factories, DABs DABs related issues software-factory PR produced by the software factories task

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants