Skip to content

Wait for grants and permissions before creating pipelines - #7142

Open
andrewnester wants to merge 9 commits into
mainfrom
fix/4573
Open

Wait for grants and permissions before creating pipelines#7142
andrewnester wants to merge 9 commits into
mainfrom
fix/4573

Conversation

@andrewnester

@andrewnester andrewnester commented Oct 21, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Changes

  • Pipelines now wait for the grants and permissions of the resources they refer to, directly or through lifecycle.depends_on. The waits are added as lifecycle.depends_on entries by CapturePipelineDependencies, which replaces CaptureGrantsDependencies.
  • Only pipelines get implicit entries. Other resources get none. The cycle check still considers the whole graph, so a dependency that would create a cycle is skipped.
  • The mutator writes through the typed Pipeline.Lifecycle.DependsOn, in sorted pipeline order with grants before permissions.

Why

Creating a pipeline fails when its run_as principal has not yet been granted access to the schema in the same bundle. Ordering the pipeline after the grants and permissions it depends on fixes this.

Fixes #4573

Tests

  • Unit: TestCapturePipelineDependencies covers pipeline-to-schema grants, the schema-not-catalog case, pipeline-to-job permissions, non-pipeline resources getting nothing, empty grants and permissions, a null pipeline entry not panicking, cycles through apps and permissions, lifecycle.depends_on, and idempotence across repeated runs.
  • Acceptance bundle/resources/grants/schemas/pipeline_run_as: the bundle plan -o json depends_on assertion is the regression guard. The deploy step's request summary shows the schema grants PATCH before the pipeline POST.
  • Acceptance bundle/resources/permissions/pipelines/depends_on: a pipeline that refers to a job waits for the job's permissions.

This pull request and its description were written by Isaac.

@andrewnester
andrewnester requested review from a team as code owners October 21, 2026 10:28
@github-actions github-actions Bot added the DABs DABs related issues label Oct 21, 2026
@github-actions

github-actions Bot commented Oct 21, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Commit: 69173453f | Workflow run

Regression test report

Selected 2 changed tests (limit=20, 0 not selected)

4 tests: regression=4 unreleased=0 coverage=0 failing=0
test branch main (e3aeacc) latest (v1.26.0)
TestAccept/bundle/resources/grants/schemas/pipeline_run_as/DMS= ✅ ❌ ❌
TestAccept/bundle/resources/grants/schemas/pipeline_run_as/DMS=true ✅ ❌ ❌
TestAccept/bundle/resources/permissions/pipelines/depends_on/DMS= ✅ ❌ ❌
TestAccept/bundle/resources/permissions/pipelines/depends_on/DMS=true ✅ ❌ ❌
TestAccept/bundle/resources/grants/schemas/pipeline_run_as/DMS=: branch ✅ | main (e3aeacc) ❌ | latest (v1.26.0) ❌

main (e3aeacc):

acceptance_test.go:1374: Diff:
        --- bundle/resources/grants/schemas/pipeline_run_as/output.txt
        +++ /tmp/TestAcceptbundleresourcesgrantsschemaspipeline_run_asDMS=1065000650/002/output.txt
        @@ -7,11 +7,7 @@
             "pipeline_catalog": null
           },
           "pipelines": {
        -    "load": [
        -      {
        -        "resource": "schemas.src.grants"
        -      }
        -    ]
        +    "load": null
           },
           "schemas": {
             "src": null
        @@ -37,10 +33,6 @@
             {
               "node": "resources.schemas.src",
               "label": "${resources.schemas.src.name}"
        -    },
        -    {
        -      "node": "resources.schemas.src.grants",
        -      "label": "depends_on"
             }
           ],
           "resources.schemas.src": [

latest (v1.26.0): same output as main (e3aeacc)

TestAccept/bundle/resources/grants/schemas/pipeline_run_as/DMS=true: branch ✅ | main (e3aeacc) ❌ | latest (v1.26.0) ❌

main (e3aeacc):

acceptance_test.go:1374: Diff:
        --- bundle/resources/grants/schemas/pipeline_run_as/output.txt
        +++ /tmp/TestAcceptbundleresourcesgrantsschemaspipeline_run_asDMS=t2172647530/002/output.txt
        @@ -7,11 +7,7 @@
             "pipeline_catalog": null
           },
           "pipelines": {
        -    "load": [
        -      {
        -        "resource": "schemas.src.grants"
        -      }
        -    ]
        +    "load": null
           },
           "schemas": {
             "src": null
        @@ -37,10 +33,6 @@
             {
               "node": "resources.schemas.src",
               "label": "${resources.schemas.src.name}"
        -    },
        -    {
        -      "node": "resources.schemas.src.grants",
        -      "label": "depends_on"
             }
           ],
           "resources.schemas.src": [

latest (v1.26.0):

acceptance_test.go:1374: Diff:
        --- bundle/resources/grants/schemas/pipeline_run_as/output.txt
        +++ /tmp/TestAcceptbundleresourcesgrantsschemaspipeline_run_asDMS=t2297129822/002/output.txt
        @@ -7,11 +7,7 @@
             "pipeline_catalog": null
           },
           "pipelines": {
        -    "load": [
        -      {
        -        "resource": "schemas.src.grants"
        -      }
        -    ]
        +    "load": null
           },
           "schemas": {
             "src": null
        @@ -37,10 +33,6 @@
             {
               "node": "resources.schemas.src",
               "label": "${resources.schemas.src.name}"
        -    },
        -    {
        -      "node": "resources.schemas.src.grants",
        -      "label": "depends_on"
             }
           ],
           "resources.schemas.src": [
        @@ -69,8 +61,8 @@
         
         >>> print_requests.py --summary //unity-catalog/schemas //unity-catalog/permissions/schema //pipelines
         POST /api/2.1/unity-catalog/schemas
        -PATCH /api/2.1/unity-catalog/permissions/schema/catalog_grants_pipeline_[UNIQUE_NAME].src
         POST /api/2.0/pipelines
        +PATCH /api/2.1/unity-catalog/permissions/schema/catalog_grants_pipeline_[UNIQUE_NAME].src
         
         >>> [CLI] bundle plan
         Plan: 0 to add, 0 to change, 0 to delete, 5 unchanged
        
FAIL
TestAccept/bundle/resources/permissions/pipelines/depends_on/DMS=: branch ✅ | main (e3aeacc) ❌ | latest (v1.26.0) ❌

main (e3aeacc):

acceptance_test.go:1374: 
        	Error Trace:	/home/runner/work/cli/cli/libs/testdiff/testdiff.go:22
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1374
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1115
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:678
        	Error:      	Not equal: 
        	            	expected: "\n=== A pipeline that refers to a job waits for the job's permissions\n\n>>> [CLI] bundle plan -o json\n{\n  \"resources.jobs.notify\": null,\n  \"resources.jobs.notify.permissions\": [\n    {\n      \"node\": \"resources.jobs.notify\",\n      \"label\": \"${resources.jobs.notify.id}\"\n    }\n  ],\n  \"resources.pipelines.load\": [\n    {\n      \"node\": \"resources.jobs.notify\",\n      \"label\": \"${resources.jobs.notify.name}\"\n    },\n    {\n      \"node\": \"resources.jobs.notify.permissions\",\n      \"label\": \"depends_on\"\n    }\n  ]\n}\n"
        	            	actual  : "\n=== A pipeline that refers to a job waits for the job's permissions\n\n>>> [CLI] bundle plan -o json\n{\n  \"resources.jobs.notify\": null,\n  \"resources.jobs.notify.permissions\": [\n    {\n      \"node\": \"resources.jobs.notify\",\n      \"label\": \"${resources.jobs.notify.id}\"\n    }\n  ],\n  \"resources.pipelines.load\": [\n    {\n      \"node\": \"resources.jobs.notify\",\n      \"label\": \"${resources.jobs.notify.name}\"\n    }\n  ]\n}\n"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -16,6 +16,2 @@
        	            	       "label": "${resources.jobs.notify.name}"
        	            	-    },
        	            	-    {
        	            	-      "node": "resources.jobs.notify.permissions",
        	            	-      "label": "depends_on"
        	            	     }
        	Test:       	TestAccept/bundle/resources/permissions/pipelines/depends_on/DMS=
        	Messages:   	bundle/resources/permissions/pipelines/depends_on/output.txt vs /tmp/TestAcceptbundleresourcespermissionspipelinesdepends_onDMS1144662279/002/output.txt

latest (v1.26.0): same output as main (e3aeacc)

TestAccept/bundle/resources/permissions/pipelines/depends_on/DMS=true: branch ✅ | main (e3aeacc) ❌ | latest (v1.26.0) ❌

main (e3aeacc):

acceptance_test.go:1374: 
        	Error Trace:	/home/runner/work/cli/cli/libs/testdiff/testdiff.go:22
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1374
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1115
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:678
        	Error:      	Not equal: 
        	            	expected: "\n=== A pipeline that refers to a job waits for the job's permissions\n\n>>> [CLI] bundle plan -o json\n{\n  \"resources.jobs.notify\": null,\n  \"resources.jobs.notify.permissions\": [\n    {\n      \"node\": \"resources.jobs.notify\",\n      \"label\": \"${resources.jobs.notify.id}\"\n    }\n  ],\n  \"resources.pipelines.load\": [\n    {\n      \"node\": \"resources.jobs.notify\",\n      \"label\": \"${resources.jobs.notify.name}\"\n    },\n    {\n      \"node\": \"resources.jobs.notify.permissions\",\n      \"label\": \"depends_on\"\n    }\n  ]\n}\n"
        	            	actual  : "\n=== A pipeline that refers to a job waits for the job's permissions\n\n>>> [CLI] bundle plan -o json\n{\n  \"resources.jobs.notify\": null,\n  \"resources.jobs.notify.permissions\": [\n    {\n      \"node\": \"resources.jobs.notify\",\n      \"label\": \"${resources.jobs.notify.id}\"\n    }\n  ],\n  \"resources.pipelines.load\": [\n    {\n      \"node\": \"resources.jobs.notify\",\n      \"label\": \"${resources.jobs.notify.name}\"\n    }\n  ]\n}\n"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -16,6 +16,2 @@
        	            	       "label": "${resources.jobs.notify.name}"
        	            	-    },
        	            	-    {
        	            	-      "node": "resources.jobs.notify.permissions",
        	            	-      "label": "depends_on"
        	            	     }
        	Test:       	TestAccept/bundle/resources/permissions/pipelines/depends_on/DMS=true
        	Messages:   	bundle/resources/permissions/pipelines/depends_on/output.txt vs /tmp/TestAcceptbundleresourcespermissionspipelinesdepends_onDMS996845158/002/output.txt
FAIL

latest (v1.26.0):

acceptance_test.go:1374: 
        	Error Trace:	/home/runner/work/cli/cli/libs/testdiff/testdiff.go:22
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1374
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1115
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:678
        	Error:      	Not equal: 
        	            	expected: "\n=== A pipeline that refers to a job waits for the job's permissions\n\n>>> [CLI] bundle plan -o json\n{\n  \"resources.jobs.notify\": null,\n  \"resources.jobs.notify.permissions\": [\n    {\n      \"node\": \"resources.jobs.notify\",\n      \"label\": \"${resources.jobs.notify.id}\"\n    }\n  ],\n  \"resources.pipelines.load\": [\n    {\n      \"node\": \"resources.jobs.notify\",\n      \"label\": \"${resources.jobs.notify.name}\"\n    },\n    {\n      \"node\": \"resources.jobs.notify.permissions\",\n      \"label\": \"depends_on\"\n    }\n  ]\n}\n"
        	            	actual  : "\n=== A pipeline that refers to a job waits for the job's permissions\n\n>>> [CLI] bundle plan -o json\n{\n  \"resources.jobs.notify\": null,\n  \"resources.jobs.notify.permissions\": [\n    {\n      \"node\": \"resources.jobs.notify\",\n      \"label\": \"${resources.jobs.notify.id}\"\n    }\n  ],\n  \"resources.pipelines.load\": [\n    {\n      \"node\": \"resources.jobs.notify\",\n      \"label\": \"${resources.jobs.notify.name}\"\n    }\n  ]\n}\n"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -16,6 +16,2 @@
        	            	       "label": "${resources.jobs.notify.name}"
        	            	-    },
        	            	-    {
        	            	-      "node": "resources.jobs.notify.permissions",
        	            	-      "label": "depends_on"
        	            	     }
        	Test:       	TestAccept/bundle/resources/permissions/pipelines/depends_on/DMS=true
        	Messages:   	bundle/resources/permissions/pipelines/depends_on/output.txt vs /tmp/TestAcceptbundleresourcespermissionspipelinesdepends_onDMS1293139548/002/output.txt

@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Oct 21, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

✅ Integration test report

Commit: 6917345

Run: 43745771570

Env ✅​pass 🙈​skip Time
✅​ aws linux-2core-8gb 271 17 3:49
✅​ aws-windows-latest-4core-16gb 273 15 3:24
✅​ azure linux-2core-8gb 270 17 3:32
✅​ azure-windows-latest-4core-16gb 272 15 3:25
✅​ gcp linux-2core-8gb 271 17 3:45
✅​ gcp-windows-latest-4core-16gb 273 15 3:34
Top 3 slowest tests (at least 2 minutes):
duration env testname
3:29 gcp-windows-latest-4core-16gb TestAccept
3:20 azure-windows-latest-4core-16gb TestAccept
3:19 aws-windows-latest-4core-16gb TestAccept

denik
denik reviewed Oct 21, 2026
View reviewed changes
Comment thread bundle/direct/bundle_plan.go Resolved
@andrewnester
andrewnester requested a review from denik October 22, 2026 08:47
@andrewnester
andrewnester changed the base branch from main to depends-on-grants November 10, 2026 08:35
@andrewnester
andrewnester added this pull request to stack #7436 November 10, 2026 08:35
Base automatically changed from depends-on-grants to main November 10, 2026 15:09
@andrewnester
andrewnester enabled stacked auto-merge November 10, 2026 15:16
andrewnester and others added 9 commits November 15, 2026 18:05
@andrewnester
Wait for grants to provision as part of depends on
72a3c78
@andrewnester
fix changelog
9cb0a9c
@andrewnester Isaac
Capture grants dependencies as lifecycle.depends_on
8432652 
A resource that depends on another resource, through a reference or through
lifecycle.depends_on, now also depends on that resource's grants. The new
CaptureGrantsDependencies mutator records this as a lifecycle.depends_on entry
targeting the grants, e.g. schemas.src.grants, instead of adding edges to the
deployment plan after the fact. A dependency that would create a cycle is
skipped, e.g. when a schema grants privileges to the service principal of an
app that refers to the schema.

This requires lifecycle.depends_on to accept grants and permissions targets,
which lands separately on the depends-on-grants branch.

Co-authored-by: Isaac <no-reply@databricks.com>
@andrewnester Isaac
Remove leftover addGrantsDependencies call from direct plan
5f8f954 
The previous commit moved grants dependencies into the CaptureGrantsDependencies
mutator, which records them as lifecycle.depends_on entries. It deleted the
addGrantsDependencies helper but left its call in makePlan, which broke the
build of bundle/direct and every job that compiles it.

Remove the call so the plan relies only on the mutator output.

Co-authored-by: Isaac <no-reply@databricks.com>
@andrewnester Isaac
Rename CaptureGrantsDependencies to CapturePipelineDependencies
d7beef6 
The mutator is about to be limited to pipelines, so name it after them. This
commit only renames the mutator and its file; the behaviour changes next.

Co-authored-by: Isaac <no-reply@databricks.com>
@andrewnester Isaac
Only add grants and permissions depends_on for pipelines
c11ba4a 
A pipeline cannot be created until the privileges it depends on are in place, for
example its run_as principal's CREATE TABLE on the schema it writes to. Limit the
implicit lifecycle.depends_on entries to pipelines, and make each pipeline wait for
the grants and permissions of the resources it refers to, directly or through
lifecycle.depends_on, when those exist. Entries are written through the typed
config. A dependency that would create a cycle is still skipped, and the cycle
check considers the whole graph.

Co-authored-by: Isaac <no-reply@databricks.com>
@andrewnester Isaac
Cover pipeline depends_on in acceptance tests and changelog
bca6c4b 
The pipeline_run_as test checks the implicit depends_on in validate and plan and
the schema request order on deploy. A new permissions test checks that a pipeline
waits for the permissions of a job it refers to. The migrate test is restored to
origin/main, since only pipelines get implicit entries now.

Co-authored-by: Isaac <no-reply@databricks.com>
@andrewnester Isaac
Drop the run_as privilege check from the pipeline fake
8ecdfb8 
The fake's create-time 403 only fires in some runs without the mutator, so the
plan assertion in the pipeline_run_as acceptance test is the regression guard.
Revert the fake to origin/main.

Co-authored-by: Isaac <no-reply@databricks.com>
@andrewnester Isaac
Format the pipeline dependency unit tests
6917345 
Co-authored-by: Isaac <no-reply@databricks.com>
@andrewnester andrewnester changed the title Wait for grants to provision as part of depends on Wait for grants and permissions before creating pipelines Nov 15, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@pietern pietern Awaiting requested review from pietern
@janniklasrose janniklasrose Awaiting requested review from janniklasrose
@denik denik Awaiting requested review from denik

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

DABs DABs related issues

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Schema-Grants need to run before pipeline deployment

3 participants

@andrewnester @eng-dev-ecosystem-bot @denik