Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .nextchanges/notable-changes/pipeline-refresh-by-file.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
* Add `file:<path>` selectors to `pipelines run` and pipeline `bundle run`. `--refresh` and `--full-refresh` now use space-separated lists, such as `--refresh "orders payments"`, instead of comma-separated lists. Single names containing spaces now require backticks, e.g. ``--refresh 'table:`order details`'``. ([#7184](https://github.com/databricks/cli/pull/7184))
27 changes: 22 additions & 5 deletions acceptance/bundle/help/bundle-run/output.txt
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,23 @@ parameter names.
If the specified job does not use job parameters and the job has a Python file
task or a Python wheel task, the second example applies.

For pipeline runs, --refresh and --full-refresh accept:
Tables: orders or table:orders
Source files: file:./src/pipeline.py (file: is required)

Put multiple selectors in quotes, or repeat the flag:
--refresh "orders payments"
--refresh "file:./src/a.py file:./src/b.py"
Comment thread
chenyuem-db marked this conversation as resolved.
Resolved
--refresh table:orders --refresh table:payments

For a path containing spaces:
--refresh 'file:`./src/my pipeline.py`'

Each refresh mode accepts either tables or files.
Local file paths are relative to the current directory and map to deployed files.
Absolute /Workspace/ paths are also accepted.
Deploy local changes before running.

---------------------------------------------------------

You can also use the bundle run command to execute scripts / commands in the same
Expand Down Expand Up @@ -54,11 +71,11 @@ Job Task Flags:
--sql-params stringToString A map from keys to values for jobs with SQL tasks. (default [])

Pipeline Flags:
--full-refresh strings List of tables to reset and recompute.
--full-refresh-all Perform a full graph reset and recompute.
--refresh strings List of tables to update.
--refresh-all Perform a full graph update.
--validate-only Perform an update to validate graph correctness.
--full-refresh selectors Space-separated table:<name> or file:<path> selectors to reset and recompute. Quote lists or repeat the flag.
--full-refresh-all Perform a full graph reset and recompute.
--refresh selectors Space-separated table:<name> or file:<path> selectors to update. Quote lists or repeat the flag.
--refresh-all Perform a full graph update.
--validate-only Perform an update to validate graph correctness.

Flags:
-h, --help help for run
Expand Down
56 changes: 53 additions & 3 deletions acceptance/bundle/run/refresh-flags/output.txt
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ Resources: 1 created, 0 changed, 0 deleted, 0 unchanged

=== Running pipeline with --refresh flag and specific tables

>>> [CLI] bundle run my_pipeline --refresh table1,table2
>>> [CLI] bundle run my_pipeline --refresh table1 table2
Update URL: [DATABRICKS_URL]/#joblist/pipelines/[UUID][0]/updates/[UUID][1]

Update ID: [UUID][1]
Expand Down Expand Up @@ -42,7 +42,7 @@ Update ID: [UUID][2]

=== Running pipeline with --full-refresh flag and specific tables

>>> [CLI] bundle run my_pipeline --full-refresh table1,table2
>>> [CLI] bundle run my_pipeline --full-refresh table1 table2
Update URL: [DATABRICKS_URL]/#joblist/pipelines/[UUID][0]/updates/[UUID][3]

Update ID: [UUID][3]
Expand All @@ -61,7 +61,7 @@ Update ID: [UUID][3]

=== Running pipeline with --full-refresh flag and --refresh flag

>>> [CLI] bundle run my_pipeline --full-refresh table1,table2 --refresh table3,table4
>>> [CLI] bundle run my_pipeline --full-refresh table1 table2 --refresh table3 table4
Update URL: [DATABRICKS_URL]/#joblist/pipelines/[UUID][0]/updates/[UUID][4]

Update ID: [UUID][4]
Expand All @@ -81,3 +81,53 @@ Update ID: [UUID][4]
]
}
}

=== Repeated refresh flags

>>> [CLI] bundle run my_pipeline --refresh table1 --refresh table:table2
Update URL: [DATABRICKS_URL]/#joblist/pipelines/[UUID][0]/updates/[UUID][5]

Update ID: [UUID][5]

>>> print_requests.py //pipelines
{
"method": "POST",
"path": "/api/2.0/pipelines/[UUID][0]/updates",
"body": {
"refresh_selection": [
"table1",
"table2"
]
}
}

=== Quoted table selectors

>>> [CLI] bundle run my_pipeline --refresh table:`table1` `table2`
Update URL: [DATABRICKS_URL]/#joblist/pipelines/[UUID][0]/updates/[UUID][6]

Update ID: [UUID][6]

>>> print_requests.py //pipelines
{
"method": "POST",
"path": "/api/2.0/pipelines/[UUID][0]/updates",
"body": {
"refresh_selection": [
"`table1`",
"`table2`"
]
}
}

=== Invalid refresh syntax is rejected before restart
>>> musterr [CLI] bundle run my_pipeline --refresh table1,table2 --restart --no-wait
Error: invalid argument "table1,table2" for "--refresh" flag: comma-separated selectors are no longer supported; separate selectors with spaces inside quotes, or repeat the flag; use backticks for literal commas
>>> musterr [CLI] bundle run my_pipeline --full-refresh table1,table2 --restart --no-wait
Error: invalid argument "table1,table2" for "--full-refresh" flag: comma-separated selectors are no longer supported; separate selectors with spaces inside quotes, or repeat the flag; use backticks for literal commas
>>> musterr [CLI] bundle run my_pipeline --refresh table:`table1 --restart --no-wait
Error: invalid argument "table:`table1" for "--refresh" flag: unclosed backtick in refresh selector
>>> musterr [CLI] bundle run my_pipeline --refresh --restart --no-wait
Error: invalid argument "" for "--refresh" flag: expected at least one table or file selector

>>> print_requests.py //pipelines
22 changes: 19 additions & 3 deletions acceptance/bundle/run/refresh-flags/script
Original file line number Diff line number Diff line change
Expand Up @@ -2,17 +2,33 @@ trace $CLI bundle deploy
rm "$OUT_REQUESTS"

title "Running pipeline with --refresh flag and specific tables"
trace $CLI bundle run my_pipeline --refresh table1,table2
trace $CLI bundle run my_pipeline --refresh "table1 table2"
trace print_requests.py //pipelines

title "Running pipeline with --full-refresh-all flag"
trace $CLI bundle run my_pipeline --full-refresh-all
trace print_requests.py //pipelines

title "Running pipeline with --full-refresh flag and specific tables"
trace $CLI bundle run my_pipeline --full-refresh table1,table2
trace $CLI bundle run my_pipeline --full-refresh "table1 table2"
trace print_requests.py //pipelines

title "Running pipeline with --full-refresh flag and --refresh flag"
trace $CLI bundle run my_pipeline --full-refresh table1,table2 --refresh table3,table4
trace $CLI bundle run my_pipeline --full-refresh "table1 table2" --refresh "table3 table4"
trace print_requests.py //pipelines

title "Repeated refresh flags"
trace $CLI bundle run my_pipeline --refresh table1 --refresh table:table2
trace print_requests.py //pipelines

title "Quoted table selectors"
trace $CLI bundle run my_pipeline --refresh 'table:`table1` `table2`'
trace print_requests.py //pipelines

title "Invalid refresh syntax is rejected before restart"
trace musterr $CLI bundle run my_pipeline --refresh table1,table2 --restart --no-wait 2>&1 | grep -E '^(>>> |Error:)'
trace musterr $CLI bundle run my_pipeline --full-refresh table1,table2 --restart --no-wait 2>&1 | grep -E '^(>>> |Error:)'
trace musterr $CLI bundle run my_pipeline --refresh 'table:`table1' --restart --no-wait 2>&1 | grep -E '^(>>> |Error:)'
trace musterr $CLI bundle run my_pipeline --refresh '' --restart --no-wait 2>&1 | grep -E '^(>>> |Error:)'
touch "$OUT_REQUESTS"
trace print_requests.py //pipelines
21 changes: 20 additions & 1 deletion acceptance/pipelines/run/refresh-flags/databricks.yml
Original file line number Diff line number Diff line change
@@ -1,10 +1,29 @@
bundle:
name: test-pipeline-run-flags

variables:
unique_name:
description: Unique name for test resources

workspace:
root_path: ~/.bundle/${bundle.name}/${var.unique_name}

presets:
pipelines_development: true

resources:
schemas:
test_schema:
name: pipeline_refresh_${var.unique_name}
catalog_name: main
pipelines:
my_pipeline:
name: test-pipeline-flags
name: test-pipeline-flags-${var.unique_name}
catalog: main
schema: ${resources.schemas.test_schema.name}
serverless: true
libraries:
- file:
path: pipeline_file.py
- notebook:
path: src/pipeline_notebook.sql
2 changes: 1 addition & 1 deletion acceptance/pipelines/run/refresh-flags/out.test.toml
Original file line number Diff line number Diff line change
@@ -1 +1 @@
Cloud = false
Cloud = true
Loading
Loading