Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .nextchanges/bundles/depends-on-grants.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
* `lifecycle.depends_on` can now target the grants or permissions of a resource, as `schemas.foo.grants`. ([#7435](https://github.com/databricks/cli/pull/7435))
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ resources:
lifecycle:
depends_on:
- resource: a.b.c
- resource: jobs.a_job.grants
Comment thread
andrewnester marked this conversation as resolved.
Resolved
# lifecycle is a separate type on job_runs.
job_runs:
a_run:
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
Error: invalid lifecycle.depends_on: no such resource: missing_for_run
at resources.job_runs.a_run.lifecycle.depends_on[0].resource
in databricks.yml:25:23
in databricks.yml:26:23

Error: invalid lifecycle.depends_on: resource "a_job" cannot depend on itself
at resources.jobs.a_job.lifecycle.depends_on[0].resource
Expand All @@ -14,10 +14,14 @@ Error: invalid lifecycle.depends_on: no such resource: a.b.c
at resources.jobs.b_job.lifecycle.depends_on[0].resource
in databricks.yml:16:23

Error: invalid lifecycle.depends_on: no such resource: jobs.a_job.grants
at resources.jobs.b_job.lifecycle.depends_on[1].resource
in databricks.yml:17:23

Name: test-bundle
Target: default
Workspace:
User: [USERNAME]
Path: /Workspace/Users/[USERNAME]/.bundle/test-bundle/default

Found 4 errors
Found 5 errors
15 changes: 14 additions & 1 deletion acceptance/bundle/lifecycle/depends-on/databricks.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3,15 +3,28 @@ bundle:

resources:
jobs:
# No references between the jobs: the order comes from lifecycle.depends_on only.
# No references between the resources: the order comes from lifecycle.depends_on only.
a_job:
name: a_job
lifecycle:
depends_on:
- resource: z_job
- resource: pipelines.z_pipeline
- resource: jobs.z_job.permissions
- resource: schemas.z_schema.grants
z_job:
name: z_job
permissions:
- level: CAN_VIEW
user_name: viewer@example.com
pipelines:
z_pipeline:
name: z_pipeline
schemas:
z_schema:
catalog_name: main
name: z_schema
grants:
- principal: users
privileges:
- USE_SCHEMA
36 changes: 32 additions & 4 deletions acceptance/bundle/lifecycle/depends-on/output.txt
Original file line number Diff line number Diff line change
Expand Up @@ -7,25 +7,49 @@
"node": "resources.jobs.z_job",
"label": "depends_on"
},
{
"node": "resources.jobs.z_job.permissions",
"label": "depends_on"
},
{
"node": "resources.pipelines.z_pipeline",
"label": "depends_on"
},
{
"node": "resources.schemas.z_schema.grants",
"label": "depends_on"
}
],
"resources.jobs.z_job": null,
"resources.pipelines.z_pipeline": null
"resources.jobs.z_job.permissions": [
{
"node": "resources.jobs.z_job",
"label": "${resources.jobs.z_job.id}"
}
],
"resources.pipelines.z_pipeline": null,
"resources.schemas.z_schema": null,
"resources.schemas.z_schema.grants": [
{
"node": "resources.schemas.z_schema",
"label": "${resources.schemas.z_schema.id}"
}
]
}

=== Deploy creates dependencies first
>>> [CLI] bundle deploy
Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files...
Created jobs.a_job
Created jobs.z_job
Created jobs.z_job.permissions
Created pipelines.z_pipeline
Created schemas.z_schema
Created schemas.z_schema.grants
Files: 1 uploaded, 0 deleted
Resources: 3 created, 0 changed, 0 deleted, 0 unchanged
Resources: 6 created, 0 changed, 0 deleted, 0 unchanged

>>> print_requests.py //jobs //pipelines
>>> print_requests.py //jobs //pipelines //unity-catalog
POST /api/2.2/jobs/create a_job

=== Destroy deletes dependents first
Expand All @@ -34,12 +58,16 @@ The following resources will be deleted:
delete resources.jobs.a_job
delete resources.jobs.z_job
delete resources.pipelines.z_pipeline
delete resources.schemas.z_schema

This action will result in the deletion of the following UC schemas. Any underlying data may be lost:
delete resources.schemas.z_schema

This action will result in the deletion of the following Lakeflow Spark Declarative Pipelines along with the
Streaming Tables (STs) and Materialized Views (MVs) managed by them. Set 'cascade_on_destroy: false' on a pipeline to retain datasets on pipeline deletion:
delete resources.pipelines.z_pipeline

All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/test-bundle/default

Destroy: 3 deleted
Destroy: 4 deleted
a_job deleted first
4 changes: 2 additions & 2 deletions acceptance/bundle/lifecycle/depends-on/script
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,8 @@ trace $CLI bundle plan -o json | jq '.plan | map_values(.depends_on)'

title "Deploy creates dependencies first"
trace $CLI bundle deploy
# z_job and z_pipeline are unordered relative to each other, so only check that a_job is last.
trace print_requests.py //jobs //pipelines | jq -r '.method + " " + .path + " " + (.body.name // "")' | tail -n 1
# The dependencies of a_job are unordered relative to each other, so only check that a_job is last.
trace print_requests.py //jobs //pipelines //unity-catalog | jq -r '.method + " " + .path + " " + (.body.name // "")' | tail -n 1

rm -f out.requests.txt

Expand Down
19 changes: 19 additions & 0 deletions bundle/config/resources.go
Original file line number Diff line number Diff line change
Expand Up @@ -196,6 +196,25 @@ func (r *Resources) ResolveKey(ref string) (ConfigResource, string, error) {
}
}

// ResolveDependsOn returns the key of the node that a lifecycle.depends_on entry refers to:
// "resources.jobs.foo" for "foo" or "jobs.foo", and "resources.schemas.foo.grants" for the
// grants or permissions of a qualified key, as "schemas.foo.grants".
func (r *Resources) ResolveDependsOn(ref string) (string, error) {
Comment thread
andrewnester marked this conversation as resolved.
Resolved
resourceRef, sub := ref, ""
if parts := strings.Split(ref, "."); len(parts) == 3 {
if _, ok := ResourcesTypes[parts[0]+"."+parts[2]]; !ok {
return "", fmt.Errorf("no such resource: %s", ref)
}
resourceRef, sub = parts[0]+"."+parts[1], "."+parts[2]
}

res, key, err := r.ResolveKey(resourceRef)
if err != nil {
return "", err
}
return "resources." + res.ResourceDescription().PluralName + "." + key + sub, nil
}

func (r *Resources) HasInternalResources() bool {
return len(r.Snapshots) > 0
}
Expand Down
2 changes: 1 addition & 1 deletion bundle/config/resources/lifecycle.go
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ type Lifecycle struct {

// DependsOn is a single lifecycle.depends_on entry.
type DependsOn struct {
// Key of the resource to depend on: "foo" or "jobs.foo".
// Key of the resource to depend on, "foo" or "jobs.foo", or of its grants or permissions, "schemas.foo.grants".
Resource string `json:"resource"`
}

Expand Down
38 changes: 38 additions & 0 deletions bundle/config/resources_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -561,3 +561,41 @@ func TestResolveKey(t *testing.T) {
})
}
}

func TestResolveDependsOn(t *testing.T) {
r := &Resources{
Jobs: map[string]*resources.Job{"foo": {}, "thing": {}},
Pipelines: map[string]*resources.Pipeline{"thing": {}},
Schemas: map[string]*resources.Schema{"bar": {}},
}

tests := []struct {
ref string
want string
wantErr string
}{
{ref: "foo", want: "resources.jobs.foo"},
{ref: "jobs.foo", want: "resources.jobs.foo"},
{ref: "schemas.bar.grants", want: "resources.schemas.bar.grants"},
{ref: "jobs.foo.permissions", want: "resources.jobs.foo.permissions"},
{ref: "missing", wantErr: "no such resource: missing"},
{ref: "thing", wantErr: "ambiguous resource: thing (can resolve to jobs.thing, pipelines.thing); use a qualified name to disambiguate"},
{ref: "jobs.foo.grants", wantErr: "no such resource: jobs.foo.grants"},
{ref: "schemas.missing.grants", wantErr: "no such resource: schemas.missing"},
{ref: "bar.grants", wantErr: "no such resource: bar.grants"},
{ref: "a.b.c", wantErr: "no such resource: a.b.c"},
{ref: "a.b.c.d", wantErr: "no such resource: a.b.c.d"},
}

for _, tc := range tests {
t.Run(tc.ref, func(t *testing.T) {
got, err := r.ResolveDependsOn(tc.ref)
if tc.wantErr != "" {
assert.EqualError(t, err, tc.wantErr)
return
}
require.NoError(t, err)
assert.Equal(t, tc.want, got)
})
}
}
8 changes: 4 additions & 4 deletions bundle/config/validate/lifecycle_depends_on.go
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ import (
type lifecycleDependsOn struct{}

// LifecycleDependsOn validates that every lifecycle.depends_on entry refers to an existing
// resource other than the one declaring it.
// resource other than the one declaring it, or to the grants or permissions of a resource.
func LifecycleDependsOn() bundle.Mutator {
return &lifecycleDependsOn{}
}
Expand All @@ -34,10 +34,10 @@ func (*lifecycleDependsOn) Apply(_ context.Context, b *bundle.Bundle) diag.Diagn
continue
}

self := group.Description.PluralName + "." + key
self := "resources." + group.Description.PluralName + "." + key
for i, dep := range resource.GetLifecycle().GetDependsOn() {
target, targetKey, err := b.Config.Resources.ResolveKey(dep.Resource)
if err == nil && target.ResourceDescription().PluralName+"."+targetKey == self {
target, err := b.Config.Resources.ResolveDependsOn(dep.Resource)
if err == nil && target == self {
err = fmt.Errorf("resource %q cannot depend on itself", dep.Resource)
}
if err != nil {
Expand Down
3 changes: 1 addition & 2 deletions bundle/direct/bundle_plan.go
Original file line number Diff line number Diff line change
Expand Up @@ -1464,11 +1464,10 @@ func LifecycleDependsOn(configRoot *config.Root, inputConfig any) ([]deployplan.

var result []deployplan.DependsOnEntry
for _, dep := range resource.GetLifecycle().GetDependsOn() {
res, key, err := configRoot.Resources.ResolveKey(dep.Resource)
target, err := configRoot.Resources.ResolveDependsOn(dep.Resource)
if err != nil {
return nil, fmt.Errorf("lifecycle.depends_on: %w", err)
}
target := "resources." + res.ResourceDescription().PluralName + "." + key
result = append(result, deployplan.DependsOnEntry{Node: target, Label: lifecycleDependsOnLabel})
}
return result, nil
Expand Down
2 changes: 1 addition & 1 deletion bundle/internal/schema/annotations.yml
Original file line number Diff line number Diff line change
Expand Up @@ -1206,7 +1206,7 @@ resources:
"$fields":
"resource":
"description": |-
Key of the resource to depend on, as `foo` or `jobs.foo`.
Key of the resource to depend on, as `foo` or `jobs.foo`, or of its grants or permissions, as `schemas.foo.grants`.
"prevent_destroy":
"description": |-
Lifecycle setting to prevent the resource from being destroyed.
Expand Down
2 changes: 1 addition & 1 deletion bundle/schema/jsonschema.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 2 additions & 2 deletions python/databricks/bundles/alerts/_models/depends_on.py

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 2 additions & 2 deletions python/databricks/bundles/apps/_models/depends_on.py

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 2 additions & 2 deletions python/databricks/bundles/catalogs/_models/depends_on.py

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 2 additions & 2 deletions python/databricks/bundles/clusters/_models/depends_on.py

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 2 additions & 2 deletions python/databricks/bundles/dashboards/_models/depends_on.py

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading
Loading