Skip to content

Detect remote permission changes on unmanaged resources in config-remote-sync - #7536

Open
ilyakuz-db wants to merge 5 commits into
mainfrom
config-remote-sync-unmanaged-permissions
Open

ilyakuz-db wants to merge 5 commits into
mainfrom
config-remote-sync-unmanaged-permissions

Conversation

@ilyakuz-db

@ilyakuz-db ilyakuz-db commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Changes

bundle config-remote-sync now writes remote permission changes back to the config:

  • Resources that declare no permissions: the current ACL of each deployed resource is read and added as a permissions: block. The current user's IS_OWNER/CAN_MANAGE is left out because deploy adds it (FixPermissions); an owner transferred to someone else is kept, so the next deploy does not take ownership back. This is an import: the ACL cannot tell grants that existed before the deployment from ones added since.
  • Declared permissions: remote additions, level changes and removals are written back per principal (e.g. permissions[user_name='a'].level) from the plan's existing .permissions comparison. Only entries written in the resource's own config are changed; the current user's entry and bundle-level permissions are left alone, and grants are still skipped.

The ACL read lives in configsync and uses the existing adapters, so CalculatePlan and deploy are unchanged. FixPermissions now keeps the entry that sets a principal's level as is when it does not change it. Before, it rebuilt every entry without a source location, so config-remote-sync could not find declared entries in the YAML.

Why

Permission changes made in the UI were ignored, so the synced config drifted and the next deploy reverted them.

Tests

Extended config-remote-sync/unmanaged_permissions: imports a job ACL and a pipeline ACL (the pipeline's transferred owner is kept), deploys, then remotely changes a level, adds a group and removes a user, and checks the second sync writes all three and the following deploy is a no-op. Unit tests for declared-permission changes (including an empty remote ACL), the owner filter, resource selection, ID lookup and FixPermissions locations. Ran bundle/... unit tests and the bundle/ acceptance tests locally (the ruff-based pydabs formatting checks could not run: no ruff on the machine); golangci-lint clean.

This pull request and its description were written by Isaac.

@ilyakuz-db
ilyakuz-db requested review from a team as code owners November 12, 2026 10:29
@ilyakuz-db
ilyakuz-db requested a review from rugpanov November 12, 2026 10:29
@ilyakuz-db ilyakuz-db added the ai-auto PRs made by AI loops & factories, label Nov 12, 2026
@ilyakuz-db ilyakuz-db self-assigned this Nov 12, 2026
@github-actions github-actions Bot added the DABs DABs related issues label Nov 12, 2026
@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Commit: b9f3db2e6 | Workflow run

Regression test report

Selected 3 changed tests (limit=20, 0 not selected)

6 tests: regression=2 unreleased=0 coverage=4 failing=0
test branch main (8e70c8b) latest (v1.26.0)
TestAccept/bundle/config-remote-sync/job_fields/DMS= ✅ ✅ ✅
TestAccept/bundle/config-remote-sync/job_fields/DMS=true ✅ ✅ ✅
TestAccept/bundle/config-remote-sync/pipeline_fields/DMS= ✅ ✅ ✅
TestAccept/bundle/config-remote-sync/pipeline_fields/DMS=true ✅ ✅ ✅
TestAccept/bundle/config-remote-sync/unmanaged_permissions/DMS= ✅ ❌ ❌
TestAccept/bundle/config-remote-sync/unmanaged_permissions/DMS=true ✅ ❌ ❌
TestAccept/bundle/config-remote-sync/unmanaged_permissions/DMS=: branch ✅ | main (8e70c8b) ❌ | latest (v1.26.0) ❌

main (8e70c8b):

acceptance_test.go:1374: Diff:
        --- bundle/config-remote-sync/unmanaged_permissions/output.txt
        +++ /tmp/TestAcceptbundleconfig-remote-syncunmanaged_permissionsDMS=3506946666/002/output.txt
        @@ -7,90 +7,39 @@
         === Add permissions remotely to resources that declare none
         
         === Detect and save changes
        -Detected changes in 2 resource(s):
        +No changes detected.
         
        -Resource: resources.jobs.my_job
        -  permissions: add
         
        -Resource: resources.pipelines.my_pipeline
        -  permissions: add
        -
        -
        -
         === Configuration changes
         
         >>> diff.py databricks.yml.backup databricks.yml
        ---- databricks.yml.backup
        -+++ databricks.yml
        -@@ -14,4 +14,9 @@
        -             num_workers: 1
         
        -+      permissions:
        -+        - level: CAN_MANAGE
        -+          user_name: viewer@example.com
        -+        - level: CAN_MANAGE_RUN
        -+          user_name: admin@example.com
        -   pipelines:
        -     my_pipeline:
        -@@ -20,4 +25,9 @@
        -         - notebook:
        -             path: /Users/{{workspace_user_name}}/notebook
        -+      permissions:
        -+        - level: IS_OWNER
        -+          user_name: owner@example.com
        -+        - group_name: users
        -+          level: CAN_VIEW
        -
        - targets:
        -
         === Sync again: the permissions are now in the config
         No changes detected.
         
         
         === Deploy keeps the remote owner
         Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle-[UNIQUE_NAME]/default/files...
        -Created jobs.my_job.permissions
        -Created pipelines.my_pipeline.permissions
        -Files: 1 uploaded, 0 deleted
        -Resources: 2 created, 0 changed, 0 deleted, 2 unchanged
        +Files: 0 uploaded, 0 deleted
        +Resources: 0 created, 0 changed, 0 deleted, 2 unchanged
         
         >>> [CLI] permissions get pipelines [MY_PIPELINE_ID]
        -[{"user_name":"owner@example.com","group_name":null,"level":"IS_OWNER"},{"user_name":null,"group_name":"users","level":"CAN_VIEW"},{"user_name":"[USERNAME]","group_name":null,"level":"CAN_MANAGE"}]
        +[{"user_name":"owner@example.com","group_name":null,"level":"IS_OWNER"},{"user_name":"[USERNAME]","group_name":null,"level":"CAN_MANAGE"},{"user_name":null,"group_name":"users","level":"CAN_VIEW"}]
         
         === Change a level, add and remove principals remotely
         
         === Detect and save changes
        -Detected changes in 1 resource(s):
        +No changes detected.
         
        -Resource: resources.jobs.my_job
        -  permissions[group_name='data-team']: add
        -  permissions[user_name='admin@example.com']: remove
        -  permissions[user_name='viewer@example.com'].level: replace
         
        -
        -
         === Configuration changes
         
         >>> diff.py databricks.yml.backup databricks.yml
        ---- databricks.yml.backup
        -+++ databricks.yml
        -@@ -15,8 +15,8 @@
         
        -       permissions:
        --        - level: CAN_MANAGE
        -+        - level: CAN_VIEW
        -           user_name: viewer@example.com
        --        - level: CAN_MANAGE_RUN
        --          user_name: admin@example.com
        -+        - group_name: data-team
        -+          level: CAN_MANAGE_RUN
        -   pipelines:
        -     my_pipeline:
        -
         === Deploy and sync again
         Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle-[UNIQUE_NAME]/default/files...
        -Files: 1 uploaded, 0 deleted
        -Resources: 0 created, 0 changed, 0 deleted, 4 unchanged
        +Files: 0 uploaded, 0 deleted
        +Resources: 0 created, 0 changed, 0 deleted, 2 unchanged
         No changes detected.

latest (v1.26.0): same output as main (8e70c8b)

TestAccept/bundle/config-remote-sync/unmanaged_permissions/DMS=true: branch ✅ | main (8e70c8b) ❌ | latest (v1.26.0) ❌

main (8e70c8b):

acceptance_test.go:1374: Diff:
        --- bundle/config-remote-sync/unmanaged_permissions/output.txt
        +++ /tmp/TestAcceptbundleconfig-remote-syncunmanaged_permissionsDMS=t2818340359/002/output.txt
        @@ -7,90 +7,39 @@
         === Add permissions remotely to resources that declare none
         
         === Detect and save changes
        -Detected changes in 2 resource(s):
        +No changes detected.
         
        -Resource: resources.jobs.my_job
        -  permissions: add
         
        -Resource: resources.pipelines.my_pipeline
        -  permissions: add
        -
        -
        -
         === Configuration changes
         
         >>> diff.py databricks.yml.backup databricks.yml
        ---- databricks.yml.backup
        -+++ databricks.yml
        -@@ -14,4 +14,9 @@
        -             num_workers: 1
         
        -+      permissions:
        -+        - level: CAN_MANAGE
        -+          user_name: viewer@example.com
        -+        - level: CAN_MANAGE_RUN
        -+          user_name: admin@example.com
        -   pipelines:
        -     my_pipeline:
        -@@ -20,4 +25,9 @@
        -         - notebook:
        -             path: /Users/{{workspace_user_name}}/notebook
        -+      permissions:
        -+        - level: IS_OWNER
        -+          user_name: owner@example.com
        -+        - group_name: users
        -+          level: CAN_VIEW
        -
        - targets:
        -
         === Sync again: the permissions are now in the config
         No changes detected.
         
         
         === Deploy keeps the remote owner
         Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle-[UNIQUE_NAME]/default/files...
        -Created jobs.my_job.permissions
        -Created pipelines.my_pipeline.permissions
        -Files: 1 uploaded, 0 deleted
        -Resources: 2 created, 0 changed, 0 deleted, 2 unchanged
        +Files: 0 uploaded, 0 deleted
        +Resources: 0 created, 0 changed, 0 deleted, 2 unchanged
         
         >>> [CLI] permissions get pipelines [MY_PIPELINE_ID]
        -[{"user_name":"owner@example.com","group_name":null,"level":"IS_OWNER"},{"user_name":null,"group_name":"users","level":"CAN_VIEW"},{"user_name":"[USERNAME]","group_name":null,"level":"CAN_MANAGE"}]
        +[{"user_name":"owner@example.com","group_name":null,"level":"IS_OWNER"},{"user_name":"[USERNAME]","group_name":null,"level":"CAN_MANAGE"},{"user_name":null,"group_name":"users","level":"CAN_VIEW"}]
         
         === Change a level, add and remove principals remotely
         
         === Detect and save changes
        -Detected changes in 1 resource(s):
        +No changes detected.
         
        -Resource: resources.jobs.my_job
        -  permissions[group_name='data-team']: add
        -  permissions[user_name='admin@example.com']: remove
        -  permissions[user_name='viewer@example.com'].level: replace
         
        -
        -
         === Configuration changes
         
         >>> diff.py databricks.yml.backup databricks.yml
        ---- databricks.yml.backup
        -+++ databricks.yml
        -@@ -15,8 +15,8 @@
         
        -       permissions:
        --        - level: CAN_MANAGE
        -+        - level: CAN_VIEW
        -           user_name: viewer@example.com
        --        - level: CAN_MANAGE_RUN
        --          user_name: admin@example.com
        -+        - group_name: data-team
        -+          level: CAN_MANAGE_RUN
        -   pipelines:
        -     my_pipeline:
        -
         === Deploy and sync again
         Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle-[UNIQUE_NAME]/default/files...
        -Files: 1 uploaded, 0 deleted
        -Resources: 0 created, 0 changed, 0 deleted, 4 unchanged
        +Files: 0 uploaded, 0 deleted
        +Resources: 0 created, 0 changed, 0 deleted, 2 unchanged
         No changes detected.
         
         
        
FAIL

latest (v1.26.0): same output as main (8e70c8b)

@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Collaborator

✅ Integration test report

Commit: b9f3db2

Run: 43445904717

Env 💚​RECOVERED ✅​pass 🙈​skip Time
💚​ aws linux-2core-8gb 1 276 16 8:45
💚​ aws-windows-latest-4core-16gb 1 272 16 5:59
💚​ azure linux-2core-8gb 1 275 16 9:12
💚​ azure-windows-latest-4core-16gb 1 271 16 5:12
💚​ gcp linux-2core-8gb 1 276 16 8:36
💚​ gcp-windows-latest-4core-16gb 1 272 16 7:01
Test Name aws linux-2core-8gb aws-windows-latest-4core-16gb azure linux-2core-8gb azure-windows-latest-4core-16gb gcp linux-2core-8gb gcp-windows-latest-4core-16gb
💚​ TestAccept 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
Top 4 slowest tests (at least 2 minutes):
duration env testname
3:25 gcp-windows-latest-4core-16gb TestAccept
3:19 aws-windows-latest-4core-16gb TestAccept
3:08 azure-windows-latest-4core-16gb TestAccept
2:10 gcp-windows-latest-4core-16gb TestFilerWorkspaceFilesExtensionsStat

@rugpanov
rugpanov removed their request for review November 12, 2026 13:35
@ilyakuz-db

Copy link
Copy Markdown
Contributor Author

The live check in e2-dogfood supports reading unmanaged permissions: three DAB jobs had backend ACLs but no .permissions entry in deployment state. Each had only a direct IS_OWNER grant and inherited admins: CAN_MANAGE, which the current filters exclude.

A few things to address before merging:

  • Existing permissions: we need to sync remote additions and level changes even when the bundle already declares permissions. Currently those nodes are skipped, so subsequent UI edits after the first import are ignored. Can we reuse the existing plan/state comparison for these?
  • Ownership: filtering every IS_OWNER entry can drop a different remote owner. Once we import a permissions block, FixPermissions adds the deploying identity as owner. We should preserve existing ownership.
  • Tests: extend unmanaged_permissions to change a permission level and add another principal remotely after the first import, then verify a second sync captures both. The existing output_no_changes cloud test already covers unchanged backend defaults.

For unmanaged permissions, this imports the current explicit ACL; it cannot distinguish pre-existing grants from changes made since the last deployment. That seems reasonable if documented as import behavior.

Comment thread bundle/configsync/diff.go Outdated
var result []dresources.StatePermission
for _, p := range remote.EmbeddedSlice {
isCurrentUser := p.UserName == currentUser || p.ServicePrincipalName == currentUser
if p.Level == iam.PermissionLevelIsOwner || (p.Level == iam.PermissionLevelCanManage && isCurrentUser) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This drops IS_OWNER for every principal, so an owner transferred in the UI is lost and the next deploy's FixPermissions reassigns it to the deployer. Drop only the current user's?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[AI] Fixed in b9f3db2: only the current user's IS_OWNER/CAN_MANAGE is dropped now (withoutCurrentUserManage in bundle/configsync/permissions.go), so a transferred owner is imported and the next deploy keeps it, adding CAN_MANAGE for the deployer. The pipeline in unmanaged_permissions covers it.

ilyakuz-db and others added 3 commits November 13, 2026 14:26
…keep other owners

Co-authored-by: Isaac <no-reply@databricks.com>
… permissions, keep the max-level entry location

Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
@ilyakuz-db

Copy link
Copy Markdown
Contributor Author

[AI] Fixed in b9f3db2:

  • Existing permissions: remote additions, level changes and removals are now written back per principal from the existing plan/state comparison. Only entries written in the resource's own config are changed; bundle-level ones and the current user's are left alone.
  • Ownership: only the current user's IS_OWNER/CAN_MANAGE is dropped on import, so a transferred owner is kept and FixPermissions adds CAN_MANAGE for the deployer instead.
  • Tests: unmanaged_permissions now deploys after the first import, then changes a level, adds a group and removes a user remotely; the second sync writes all three and the following deploy is a no-op.
  • Import behavior for undeclared permissions is documented in the code and the PR description.

Also moved the ACL read into configsync, so CalculatePlan keeps its signature. FixPermissions now keeps unchanged entries as is (with their source locations), which config-remote-sync needs to find declared entries in the YAML.

Co-authored-by: Isaac <no-reply@databricks.com>
@ilyakuz-db ilyakuz-db added the software-factory PR produced by the software factories task label Nov 15, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ai-auto PRs made by AI loops & factories, DABs DABs related issues software-factory PR produced by the software factories task

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants