Skip to content

Support file selectors in pipelines run - #7184

Merged
chenyuem-db merged 10 commits into
mainfrom
chenyuem-db/pipeline-refresh-by-file
Nov 12, 2026
Merged

chenyuem-db merged 10 commits into
mainfrom
chenyuem-db/pipeline-refresh-by-file

Conversation

@chenyuem-db

@chenyuem-db chenyuem-db commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Changes

Add file-based refresh to pipelines run and pipeline bundle run, bringing the UI's “Run file” workflow to the CLI.

  • Both --refresh and --full-refresh accept tables or source files.
  • Tables use orders or table:orders. Files require file:.
  • Local sources and notebooks map to their deployed Workspace paths. Absolute /Workspace/ paths also work.
  • Quote space-separated lists, or repeat the flag. Local checks run before restart; the backend checks whether files belong to the pipeline when the update starts.
  • Show selected files in the completion summary and fix progress-event fetching through the public API.
  • Reject unsupported file refreshes on continuous pipelines before --restart stops the active update.

Deploy local edits before running. Each refresh mode accepts either tables or files. The two modes can use different types.

# Refresh one table. Both forms work.
databricks pipelines run my_pipeline --refresh orders
databricks pipelines run my_pipeline --refresh table:orders

# Refresh several tables, or fully refresh them.
databricks pipelines run my_pipeline --refresh "table:orders table:payments"
databricks pipelines run my_pipeline --full-refresh "table:orders table:payments"

# Refresh several source files, or fully refresh one file.
databricks pipelines run my_pipeline --refresh "file:./src/a.py file:./src/b.py"
databricks pipelines run my_pipeline --full-refresh file:./src/a.py

# Select a deployed Workspace file directly.
databricks pipelines run my_pipeline --refresh file:/Workspace/Shared/project/src/a.py

# Repeat a flag instead of writing a list.
databricks pipelines run my_pipeline --refresh table:orders --refresh table:payments

# Combine different actions and target types.
databricks pipelines run my_pipeline \
  --refresh "table:orders table:payments" \
  --full-refresh file:./src/a.py

# Backticks preserve spaces. Single quotes protect them in Bash/zsh.
databricks pipelines run my_pipeline --refresh 'table:`order details`'
databricks pipelines run my_pipeline --refresh 'file:`./src/my pipeline.py`'

# The same selectors work with bundle run.
databricks bundle run my_pipeline --refresh "file:./src/a.py file:./src/b.py"
databricks bundle run my_pipeline --full-refresh "table:orders table:payments"

Why

Support SDP Turbo CLI project:

  • Keep one repeatable flag per action. Selectors carry their target type.
  • Require file: to avoid guessing from the local filesystem.
  • Use spaces for lists, following dbt's selector syntax. Commas mean intersection in dbt; removing comma lists leaves room for future selector operators.
  • Backticks preserve spaces and commas within a name or path. Double a backtick to escape it.
  • Keep the syntax ready for the proposed selections API. Requests still use the existing API fields.

Migration: replace --refresh orders,payments with --refresh "orders payments", or repeat the flag. The same applies to --full-refresh. Old comma lists return an error with migration guidance.

Showing resolved table names is a follow-up.

Tests

  • Affected unit tests passed after rebase. Coverage includes parsing, backtick escaping, 50-target lists, path mapping, and both publishing modes.
  • Existing pipeline/bundle refresh acceptance tests and help snapshots passed after rebase, including both DMS variants.
  • Formatting, repository checks, and lint passed.
  • Before this rebase, staging completed 16 updates and rejected 13 invalid inputs. Temporary resources were deleted.

@github-actions github-actions Bot added the DABs DABs related issues label Oct 26, 2026
@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Oct 26, 2026 •

Copy link
Copy Markdown
Collaborator

✅ Integration test report

Commit: 879c75e

Run: 43137597135

Env 💚​RECOVERED ✅​pass 🙈​skip Time
💚​ aws linux-2core-8gb 1 271 17 7:26
💚​ aws-windows-latest-4core-16gb 1 273 15 9:03
💚​ azure linux-2core-8gb 1 270 17 8:30
💚​ azure-windows-latest-4core-16gb 1 272 15 8:39
💚​ gcp linux-2core-8gb 1 271 17 7:52
💚​ gcp-windows-latest-4core-16gb 1 273 15 9:05
Test Name aws linux-2core-8gb aws-windows-latest-4core-16gb azure linux-2core-8gb azure-windows-latest-4core-16gb gcp linux-2core-8gb gcp-windows-latest-4core-16gb
💚​ TestAccept 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
Top 12 slowest tests (at least 2 minutes):
duration env testname
5:22 azure-windows-latest-4core-16gb TestAccept/pipelines/run/refresh-flags
5:10 azure linux-2core-8gb TestAccept/pipelines/run/refresh-flags
5:06 gcp-windows-latest-4core-16gb TestAccept/pipelines/run/refresh-flags
5:05 aws-windows-latest-4core-16gb TestAccept/pipelines/run/refresh-flags
4:11 aws linux-2core-8gb TestAccept/pipelines/run/refresh-flags
4:02 gcp linux-2core-8gb TestAccept
3:53 gcp-windows-latest-4core-16gb TestAccept
3:52 aws-windows-latest-4core-16gb TestAccept
3:50 gcp linux-2core-8gb TestAccept/pipelines/run/refresh-flags
3:19 azure linux-2core-8gb TestAccept
3:15 aws linux-2core-8gb TestAccept
3:12 azure-windows-latest-4core-16gb TestAccept

@chenyuem-db
chenyuem-db force-pushed the chenyuem-db/pipeline-refresh-by-file branch 2 times, most recently from 0824056 to 652cd9e Compare October 28, 2026 17:02
@github-actions

github-actions Bot commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Commit: 879c75eaa | Workflow run

Regression test report

Selected 3 changed tests (limit=20, 0 not selected)

5 tests: regression=5 unreleased=0 coverage=0 failing=0
test branch main (a0a6430) latest (v1.26.0)
TestAccept/bundle/help/bundle-run/DMS= ✅ ❌ ❌
TestAccept/bundle/help/bundle-run/DMS=true ✅ ❌ ❌
TestAccept/bundle/run/refresh-flags/DMS= ✅ ❌ ❌
TestAccept/bundle/run/refresh-flags/DMS=true ✅ ❌ ❌
TestAccept/pipelines/run/refresh-flags ✅ ❌ ❌
TestAccept/bundle/help/bundle-run/DMS=: branch ✅ | main (a0a6430) ❌ | latest (v1.26.0) ❌

main (a0a6430):

acceptance_test.go:1345: Diff:
        --- bundle/help/bundle-run/output.txt
        +++ /tmp/TestAcceptbundlehelpbundle-runDMS=2088859473/002/output.txt
        @@ -19,23 +19,6 @@
         If the specified job does not use job parameters and the job has a Python file
         task or a Python wheel task, the second example applies.
         
        -For pipeline runs, --refresh and --full-refresh accept:
        -  Tables: orders or table:orders
        -  Source files: file:./src/pipeline.py (file: is required)
        -
        -Put multiple selectors in quotes, or repeat the flag:
        -  --refresh "orders payments"
        -  --refresh "file:./src/a.py file:./src/b.py"
        -  --refresh table:orders --refresh table:payments
        -
        -For a path containing spaces:
        -  --refresh 'file:`./src/my pipeline.py`'
        -
        -Each refresh mode accepts either tables or files.
        -Local file paths are relative to the current directory and map to deployed files.
        -Absolute /Workspace/ paths are also accepted.
        -Deploy local changes before running.
        -
         ---------------------------------------------------------
         
         You can also use the bundle run command to execute scripts / commands in the same
        @@ -71,11 +54,11 @@
               --sql-params stringToString            A map from keys to values for jobs with SQL tasks. (default [])
         
         Pipeline Flags:
        -      --full-refresh selectors   Space-separated table:<name> or file:<path> selectors to reset and recompute. Quote lists or repeat the flag.
        -      --full-refresh-all         Perform a full graph reset and recompute.
        -      --refresh selectors        Space-separated table:<name> or file:<path> selectors to update. Quote lists or repeat the flag.
        -      --refresh-all              Perform a full graph update.
        -      --validate-only            Perform an update to validate graph correctness.
        +      --full-refresh strings   List of tables to reset and recompute.
        +      --full-refresh-all       Perform a full graph reset and recompute.
        +      --refresh strings        List of tables to update.
        +      --refresh-all            Perform a full graph update.
        +      --validate-only          Perform an update to validate graph correctness.
         
         Flags:
           -h, --help      help for run

latest (v1.26.0): same output as main (a0a6430)

TestAccept/bundle/help/bundle-run/DMS=true: branch ✅ | main (a0a6430) ❌ | latest (v1.26.0) ❌

main (a0a6430):

acceptance_test.go:1345: Diff:
        --- bundle/help/bundle-run/output.txt
        +++ /tmp/TestAcceptbundlehelpbundle-runDMS=true3584387019/002/output.txt
        @@ -19,23 +19,6 @@
         If the specified job does not use job parameters and the job has a Python file
         task or a Python wheel task, the second example applies.
         
        -For pipeline runs, --refresh and --full-refresh accept:
        -  Tables: orders or table:orders
        -  Source files: file:./src/pipeline.py (file: is required)
        -
        -Put multiple selectors in quotes, or repeat the flag:
        -  --refresh "orders payments"
        -  --refresh "file:./src/a.py file:./src/b.py"
        -  --refresh table:orders --refresh table:payments
        -
        -For a path containing spaces:
        -  --refresh 'file:`./src/my pipeline.py`'
        -
        -Each refresh mode accepts either tables or files.
        -Local file paths are relative to the current directory and map to deployed files.
        -Absolute /Workspace/ paths are also accepted.
        -Deploy local changes before running.
        -
         ---------------------------------------------------------
         
         You can also use the bundle run command to execute scripts / commands in the same
        @@ -71,11 +54,11 @@
               --sql-params stringToString            A map from keys to values for jobs with SQL tasks. (default [])
         
         Pipeline Flags:
        -      --full-refresh selectors   Space-separated table:<name> or file:<path> selectors to reset and recompute. Quote lists or repeat the flag.
        -      --full-refresh-all         Perform a full graph reset and recompute.
        -      --refresh selectors        Space-separated table:<name> or file:<path> selectors to update. Quote lists or repeat the flag.
        -      --refresh-all              Perform a full graph update.
        -      --validate-only            Perform an update to validate graph correctness.
        +      --full-refresh strings   List of tables to reset and recompute.
        +      --full-refresh-all       Perform a full graph reset and recompute.
        +      --refresh strings        List of tables to update.
        +      --refresh-all            Perform a full graph update.
        +      --validate-only          Perform an update to validate graph correctness.
         
         Flags:
           -h, --help      help for run

latest (v1.26.0): same output as main (a0a6430)

TestAccept/bundle/run/refresh-flags/DMS=: branch ✅ | main (a0a6430) ❌ | latest (v1.26.0) ❌

main (a0a6430):

acceptance_test.go:1345: Diff:
        --- bundle/run/refresh-flags/output.txt
        +++ /tmp/TestAcceptbundlerunrefresh-flagsDMS=3338737514/002/output.txt
        @@ -18,8 +18,7 @@
           "path": "/api/2.0/pipelines/[UUID][0]/updates",
           "body": {
             "refresh_selection": [
        -      "table1",
        -      "table2"
        +      "table1 table2"
             ]
           }
         }
        @@ -53,8 +52,7 @@
           "path": "/api/2.0/pipelines/[UUID][0]/updates",
           "body": {
             "full_refresh_selection": [
        -      "table1",
        -      "table2"
        +      "table1 table2"
             ]
           }
         }
        @@ -72,12 +70,10 @@
           "path": "/api/2.0/pipelines/[UUID][0]/updates",
           "body": {
             "full_refresh_selection": [
        -      "table1",
        -      "table2"
        +      "table1 table2"
             ],
             "refresh_selection": [
        -      "table3",
        -      "table4"
        +      "table3 table4"
             ]
           }
         }
        @@ -96,7 +90,7 @@
           "body": {
             "refresh_selection": [
               "table1",
        -      "table2"
        +      "table:table2"
             ]
           }
         }
        @@ -114,20 +108,12 @@
           "path": "/api/2.0/pipelines/[UUID][0]/updates",
           "body": {
             "refresh_selection": [
        -      "`table1`",
        -      "`table2`"
        +      "table:`table1` `table2`"
             ]
           }
         }
         
         === Invalid refresh syntax is rejected before restart
         >>> musterr [CLI] bundle run my_pipeline --refresh table1,table2 --restart --no-wait
        -Error: invalid argument "table1,table2" for "--refresh" flag: comma-separated selectors are no longer supported; separate selectors with spaces inside quotes, or repeat the flag; use backticks for literal commas
        ->>> musterr [CLI] bundle run my_pipeline --full-refresh table1,table2 --restart --no-wait
        -Error: invalid argument "table1,table2" for "--full-refresh" flag: comma-separated selectors are no longer supported; separate selectors with spaces inside quotes, or repeat the flag; use backticks for literal commas
        ->>> musterr [CLI] bundle run my_pipeline --refresh table:`table1 --restart --no-wait
        -Error: invalid argument "table:`table1" for "--refresh" flag: unclosed backtick in refresh selector
        ->>> musterr [CLI] bundle run my_pipeline --refresh  --restart --no-wait
        -Error: invalid argument "" for "--refresh" flag: expected at least one table or file selector
         
        ->>> print_requests.py //pipelines
        +Exit code: 1
        
    acceptance_test.go:1135: Test produced unexpected files:
        .databricks/.gitignore
        .databricks/bundle/default/deployment.json
        .databricks/bundle/default/resources.json
        .databricks/bundle/default/sync-snapshots/ee8d72b96c42a40d.json

latest (v1.26.0):

acceptance_test.go:1345: Diff:
        --- bundle/run/refresh-flags/output.txt
        +++ /tmp/TestAcceptbundlerunrefresh-flagsDMS=3519597812/002/output.txt
        @@ -18,8 +18,7 @@
           "path": "/api/2.0/pipelines/[UUID][0]/updates",
           "body": {
             "refresh_selection": [
        -      "table1",
        -      "table2"
        +      "table1 table2"
             ]
           }
         }
        @@ -53,8 +52,7 @@
           "path": "/api/2.0/pipelines/[UUID][0]/updates",
           "body": {
             "full_refresh_selection": [
        -      "table1",
        -      "table2"
        +      "table1 table2"
             ]
           }
         }
        @@ -72,12 +70,10 @@
           "path": "/api/2.0/pipelines/[UUID][0]/updates",
           "body": {
             "full_refresh_selection": [
        -      "table1",
        -      "table2"
        +      "table1 table2"
             ],
             "refresh_selection": [
        -      "table3",
        -      "table4"
        +      "table3 table4"
             ]
           }
         }
        @@ -96,7 +90,7 @@
           "body": {
             "refresh_selection": [
               "table1",
        -      "table2"
        +      "table:table2"
             ]
           }
         }
        @@ -114,20 +108,12 @@
           "path": "/api/2.0/pipelines/[UUID][0]/updates",
           "body": {
             "refresh_selection": [
        -      "`table1`",
        -      "`table2`"
        +      "table:`table1` `table2`"
             ]
           }
         }
         
         === Invalid refresh syntax is rejected before restart
         >>> musterr [CLI] bundle run my_pipeline --refresh table1,table2 --restart --no-wait
        -Error: invalid argument "table1,table2" for "--refresh" flag: comma-separated selectors are no longer supported; separate selectors with spaces inside quotes, or repeat the flag; use backticks for literal commas
        ->>> musterr [CLI] bundle run my_pipeline --full-refresh table1,table2 --restart --no-wait
        -Error: invalid argument "table1,table2" for "--full-refresh" flag: comma-separated selectors are no longer supported; separate selectors with spaces inside quotes, or repeat the flag; use backticks for literal commas
        ->>> musterr [CLI] bundle run my_pipeline --refresh table:`table1 --restart --no-wait
        -Error: invalid argument "table:`table1" for "--refresh" flag: unclosed backtick in refresh selector
        ->>> musterr [CLI] bundle run my_pipeline --refresh  --restart --no-wait
        -Error: invalid argument "" for "--refresh" flag: expected at least one table or file selector
         
        ->>> print_requests.py //pipelines
        +Exit code: 1
        
    acceptance_test.go:1135: Test produced unexpected files:
        .databricks/.gitignore
        .databricks/bundle/default/deployment.json
        .databricks/bundle/default/resources.json
        .databricks/bundle/default/sync-snapshots/820bdd59aa293ddd.json
TestAccept/bundle/run/refresh-flags/DMS=true: branch ✅ | main (a0a6430) ❌ | latest (v1.26.0) ❌

main (a0a6430):

acceptance_test.go:1345: Diff:
        --- bundle/run/refresh-flags/output.txt
        +++ /tmp/TestAcceptbundlerunrefresh-flagsDMS=true386075189/002/output.txt
        @@ -18,8 +18,7 @@
           "path": "/api/2.0/pipelines/[UUID][0]/updates",
           "body": {
             "refresh_selection": [
        -      "table1",
        -      "table2"
        +      "table1 table2"
             ]
           }
         }
        @@ -53,8 +52,7 @@
           "path": "/api/2.0/pipelines/[UUID][0]/updates",
           "body": {
             "full_refresh_selection": [
        -      "table1",
        -      "table2"
        +      "table1 table2"
             ]
           }
         }
        @@ -72,12 +70,10 @@
           "path": "/api/2.0/pipelines/[UUID][0]/updates",
           "body": {
             "full_refresh_selection": [
        -      "table1",
        -      "table2"
        +      "table1 table2"
             ],
             "refresh_selection": [
        -      "table3",
        -      "table4"
        +      "table3 table4"
             ]
           }
         }
        @@ -96,7 +90,7 @@
           "body": {
             "refresh_selection": [
               "table1",
        -      "table2"
        +      "table:table2"
             ]
           }
         }
        @@ -114,20 +108,12 @@
           "path": "/api/2.0/pipelines/[UUID][0]/updates",
           "body": {
             "refresh_selection": [
        -      "`table1`",
        -      "`table2`"
        +      "table:`table1` `table2`"
             ]
           }
         }
         
         === Invalid refresh syntax is rejected before restart
         >>> musterr [CLI] bundle run my_pipeline --refresh table1,table2 --restart --no-wait
        -Error: invalid argument "table1,table2" for "--refresh" flag: comma-separated selectors are no longer supported; separate selectors with spaces inside quotes, or repeat the flag; use backticks for literal commas
        ->>> musterr [CLI] bundle run my_pipeline --full-refresh table1,table2 --restart --no-wait
        -Error: invalid argument "table1,table2" for "--full-refresh" flag: comma-separated selectors are no longer supported; separate selectors with spaces inside quotes, or repeat the flag; use backticks for literal commas
        ->>> musterr [CLI] bundle run my_pipeline --refresh table:`table1 --restart --no-wait
        -Error: invalid argument "table:`table1" for "--refresh" flag: unclosed backtick in refresh selector
        ->>> musterr [CLI] bundle run my_pipeline --refresh  --restart --no-wait
        -Error: invalid argument "" for "--refresh" flag: expected at least one table or file selector
         
        ->>> print_requests.py //pipelines
        +Exit code: 1
        
    acceptance_test.go:1135: Test produced unexpected files:
        .databricks/.gitignore
        .databricks/bundle/default/deployment.json
        .databricks/bundle/default/resources.json
        .databricks/bundle/default/sync-snapshots/01971fde3bf71cd1.json

latest (v1.26.0):

acceptance_test.go:1345: Diff:
        --- bundle/run/refresh-flags/output.txt
        +++ /tmp/TestAcceptbundlerunrefresh-flagsDMS=true446035525/002/output.txt
        @@ -18,8 +18,7 @@
           "path": "/api/2.0/pipelines/[UUID][0]/updates",
           "body": {
             "refresh_selection": [
        -      "table1",
        -      "table2"
        +      "table1 table2"
             ]
           }
         }
        @@ -53,8 +52,7 @@
           "path": "/api/2.0/pipelines/[UUID][0]/updates",
           "body": {
             "full_refresh_selection": [
        -      "table1",
        -      "table2"
        +      "table1 table2"
             ]
           }
         }
        @@ -72,12 +70,10 @@
           "path": "/api/2.0/pipelines/[UUID][0]/updates",
           "body": {
             "full_refresh_selection": [
        -      "table1",
        -      "table2"
        +      "table1 table2"
             ],
             "refresh_selection": [
        -      "table3",
        -      "table4"
        +      "table3 table4"
             ]
           }
         }
        @@ -96,7 +90,7 @@
           "body": {
             "refresh_selection": [
               "table1",
        -      "table2"
        +      "table:table2"
             ]
           }
         }
        @@ -114,20 +108,12 @@
           "path": "/api/2.0/pipelines/[UUID][0]/updates",
           "body": {
             "refresh_selection": [
        -      "`table1`",
        -      "`table2`"
        +      "table:`table1` `table2`"
             ]
           }
         }
         
         === Invalid refresh syntax is rejected before restart
         >>> musterr [CLI] bundle run my_pipeline --refresh table1,table2 --restart --no-wait
        -Error: invalid argument "table1,table2" for "--refresh" flag: comma-separated selectors are no longer supported; separate selectors with spaces inside quotes, or repeat the flag; use backticks for literal commas
        ->>> musterr [CLI] bundle run my_pipeline --full-refresh table1,table2 --restart --no-wait
        -Error: invalid argument "table1,table2" for "--full-refresh" flag: comma-separated selectors are no longer supported; separate selectors with spaces inside quotes, or repeat the flag; use backticks for literal commas
        ->>> musterr [CLI] bundle run my_pipeline --refresh table:`table1 --restart --no-wait
        -Error: invalid argument "table:`table1" for "--refresh" flag: unclosed backtick in refresh selector
        ->>> musterr [CLI] bundle run my_pipeline --refresh  --restart --no-wait
        -Error: invalid argument "" for "--refresh" flag: expected at least one table or file selector
         
        ->>> print_requests.py //pipelines
        +Exit code: 1
        
    acceptance_test.go:1135: Test produced unexpected files:
        .databricks/.gitignore
        .databricks/bundle/default/deployment.json
        .databricks/bundle/default/resources.json
        .databricks/bundle/default/sync-snapshots/fbb2843224634f3a.json
TestAccept/pipelines/run/refresh-flags: branch ✅ | main (a0a6430) ❌ | latest (v1.26.0) ❌

main (a0a6430):

acceptance_test.go:1345: Diff:
        --- pipelines/run/refresh-flags/output.txt
        +++ /tmp/TestAcceptpipelinesrunrefresh-flags78034524/002/output.txt
        @@ -5,7 +5,7 @@
         Created schemas.test_schema
         Files: 3 uploaded, 0 deleted
         Resources: 2 created, 0 changed, 0 deleted, 0 unchanged
        -View your pipeline my_pipeline here: [DATABRICKS_URL]/pipelines/[UUID]?w=[WORKSPACE_ID]
        +View your pipeline my_pipeline here: [DATABRICKS_URL]/pipelines/[UUID][0]?w=[WORKSPACE_ID]
         View your schema test_schema here: [DATABRICKS_URL]/explore/data/main/pipeline_refresh_[UNIQUE_NAME]?w=[WORKSPACE_ID]
         
         === Initialize all tables with --full-refresh-all
        @@ -16,7 +16,7 @@
         >>> print_requests.py //pipelines
         {
           "method": "POST",
        -  "path": "/api/2.0/pipelines/[UUID]/updates",
        +  "path": "/api/2.0/pipelines/[UUID][0]/updates",
           "body": {
             "development": true,
             "full_refresh": true
        @@ -26,17 +26,16 @@
         === Running pipeline with --refresh flag and specific tables
         
         >>> run_pipeline --refresh table1 table2
        -• Refreshed [table1, table2]
        +• Refreshed [table1 table2]
         
         >>> print_requests.py //pipelines
         {
           "method": "POST",
        -  "path": "/api/2.0/pipelines/[UUID]/updates",
        +  "path": "/api/2.0/pipelines/[UUID][0]/updates",
           "body": {
             "development": true,
             "refresh_selection": [
        -      "table1",
        -      "table2"
        +      "table1 table2"
             ]
           }
         }
        @@ -44,17 +36,16 @@
         === Running pipeline with --full-refresh flag and specific tables
         
         >>> run_pipeline --full-refresh table1 table2
        -• Full refreshed [table1, table2]
        +• Full refreshed [table1 table2]
         
         >>> print_requests.py //pipelines
         {
           "method": "POST",
        -  "path": "/api/2.0/pipelines/[UUID]/updates",
        +  "path": "/api/2.0/pipelines/[UUID][0]/updates",
           "body": {
             "development": true,
             "full_refresh_selection": [
        -      "table1",
        -      "table2"
        +      "table1 table2"
             ]
           }
         }
        @@ -62,22 +46,20 @@
         === Running pipeline with --full-refresh flag and --refresh flag
         
         >>> run_pipeline --full-refresh table:table1 table:table2 --refresh table:table3 table:table4
        -• Refreshed [table3, table4]
        -• Full refreshed [table1, table2]
        +• Refreshed [table:table3 table:table4]
        +• Full refreshed [table:table1 table:table2]
         
         >>> print_requests.py //pipelines
         {
           "method": "POST",
        -  "path": "/api/2.0/pipelines/[UUID]/updates",
        +  "path": "/api/2.0/pipelines/[UUID][0]/updates",
           "body": {
             "development": true,
             "full_refresh_selection": [
        -      "table1",
        -      "table2"
        +      "table:table1 table:table2"
             ],
             "refresh_selection": [
        -      "table3",
        -      "table4"
        +      "table:table3 table:table4"
             ]
           }
         }
        @@ -85,21 +58,20 @@
         === Refreshing tables and fully refreshing a source file together
         
         >>> run_pipeline --refresh table:table3 table:table4 --full-refresh file:pipeline_file.py
        -• Refreshed [table3, table4]
        -• Full refreshed tables defined in [/Workspace/Users/[USERNAME]/.bundle/test-pipeline-run-flags/[UNIQUE_NAME]/files/pipeline_file.py]
        +• Refreshed [table:table3 table:table4]
        +• Full refreshed [file:pipeline_file.py]
         
         >>> print_requests.py //pipelines
         {
           "method": "POST",
        -  "path": "/api/2.0/pipelines/[UUID]/updates",
        +  "path": "/api/2.0/pipelines/[UUID][0]/updates",
           "body": {
             "development": true,
        -    "full_refresh_selection_by_path": [
        -      "/Workspace/Users/[USERNAME]/.bundle/test-pipeline-run-flags/[UNIQUE_NAME]/files/pipeline_file.py"
        +    "full_refresh_selection": [
        +      "file:pipeline_file.py"
             ],
             "refresh_selection": [
        -      "table3",
        -      "table4"
        +      "table:table3 table:table4"
             ]
           }
         }
        @@ -107,16 +71,16 @@
         === Refreshing tables defined in a source file
         
         >>> run_pipeline --refresh file:pipeline_file.py
        -• Refreshed tables defined in [/Workspace/Users/[USERNAME]/.bundle/test-pipeline-run-flags/[UNIQUE_NAME]/files/pipeline_file.py]
        +• Refreshed [file:pipeline_file.py]
         
         >>> print_requests.py //pipelines
         {
           "method": "POST",
        -  "path": "/api/2.0/pipelines/[UUID]/updates",
        +  "path": "/api/2.0/pipelines/[UUID][0]/updates",
           "body": {
             "development": true,
        -    "refresh_selection_by_path": [
        -      "/Workspace/Users/[USERNAME]/.bundle/test-pipeline-run-flags/[UNIQUE_NAME]/files/pipeline_file.py"
        +    "refresh_selection": [
        +      "file:pipeline_file.py"
             ]
           }
         }
        @@ -124,16 +82,16 @@
         === Fully refreshing tables defined in a source file
         
         >>> run_pipeline --full-refresh file:pipeline_file.py
        -• Full refreshed tables defined in [/Workspace/Users/[USERNAME]/.bundle/test-pipeline-run-flags/[UNIQUE_NAME]/files/pipeline_file.py]
        +• Full refreshed [file:pipeline_file.py]
         
         >>> print_requests.py //pipelines
         {
           "method": "POST",
        -  "path": "/api/2.0/pipelines/[UUID]/updates",
        +  "path": "/api/2.0/pipelines/[UUID][0]/updates",
           "body": {
             "development": true,
        -    "full_refresh_selection_by_path": [
        -      "/Workspace/Users/[USERNAME]/.bundle/test-pipeline-run-flags/[UNIQUE_NAME]/files/pipeline_file.py"
        +    "full_refresh_selection": [
        +      "file:pipeline_file.py"
             ]
           }
         }
        @@ -141,16 +93,16 @@
         === Refreshing a source notebook
         
         >>> run_pipeline --refresh file:src/pipeline_notebook.sql
        -• Refreshed tables defined in [/Workspace/Users/[USERNAME]/.bundle/test-pipeline-run-flags/[UNIQUE_NAME]/files/src/pipeline_notebook]
        +• Refreshed [file:src/pipeline_notebook.sql]
         
         >>> print_requests.py //pipelines
         {
           "method": "POST",
        -  "path": "/api/2.0/pipelines/[UUID]/updates",
        +  "path": "/api/2.0/pipelines/[UUID][0]/updates",
           "body": {
             "development": true,
        -    "refresh_selection_by_path": [
        -      "/Workspace/Users/[USERNAME]/.bundle/test-pipeline-run-flags/[UNIQUE_NAME]/files/src/pipeline_notebook"
        +    "refresh_selection": [
        +      "file:src/pipeline_notebook.sql"
             ]
           }
         }
        @@ -158,16 +104,16 @@
         === Refreshing an absolute workspace path
         
         >>> run_pipeline --refresh file:/Workspace/Users/[USERNAME]/.bundle/test-pipeline-run-flags/[UNIQUE_NAME]/files/pipeline_file.py
        -• Refreshed tables defined in [/Workspace/Users/[USERNAME]/.bundle/test-pipeline-run-flags/[UNIQUE_NAME]/files/pipeline_file.py]
        +• Refreshed [file:/Workspace/Users/[USERNAME]/.bundle/test-pipeline-run-flags/[UNIQUE_NAME]/files/pipeline_file.py]
         
         >>> print_requests.py //pipelines
         {
           "method": "POST",
        -  "path": "/api/2.0/pipelines/[UUID]/updates",
        +  "path": "/api/2.0/pipelines/[UUID][0]/updates",
           "body": {
             "development": true,
        -    "refresh_selection_by_path": [
        -      "/Workspace/Users/[USERNAME]/.bundle/test-pipeline-run-flags/[UNIQUE_NAME]/files/pipeline_file.py"
        +    "refresh_selection": [
        +      "file:/Workspace/Users/[USERNAME]/.bundle/test-pipeline-run-flags/[UNIQUE_NAME]/files/pipeline_file.py"
... (246 more lines truncated, raise DATABRICKS_CLI_TEST_MAX_LOG to see more)

latest (v1.26.0):

acceptance_test.go:1345: Diff:
        --- pipelines/run/refresh-flags/output.txt
        +++ /tmp/TestAcceptpipelinesrunrefresh-flags3635847236/002/output.txt
        @@ -5,7 +5,7 @@
         Created schemas.test_schema
         Files: 3 uploaded, 0 deleted
         Resources: 2 created, 0 changed, 0 deleted, 0 unchanged
        -View your pipeline my_pipeline here: [DATABRICKS_URL]/pipelines/[UUID]?w=[WORKSPACE_ID]
        +View your pipeline my_pipeline here: [DATABRICKS_URL]/pipelines/[UUID][0]?w=[WORKSPACE_ID]
         View your schema test_schema here: [DATABRICKS_URL]/explore/data/main/pipeline_refresh_[UNIQUE_NAME]?w=[WORKSPACE_ID]
         
         === Initialize all tables with --full-refresh-all
        @@ -16,7 +16,7 @@
         >>> print_requests.py //pipelines
         {
           "method": "POST",
        -  "path": "/api/2.0/pipelines/[UUID]/updates",
        +  "path": "/api/2.0/pipelines/[UUID][0]/updates",
           "body": {
             "development": true,
             "full_refresh": true
        @@ -26,17 +26,16 @@
         === Running pipeline with --refresh flag and specific tables
         
         >>> run_pipeline --refresh table1 table2
        -• Refreshed [table1, table2]
        +• Refreshed [table1 table2]
         
         >>> print_requests.py //pipelines
         {
           "method": "POST",
        -  "path": "/api/2.0/pipelines/[UUID]/updates",
        +  "path": "/api/2.0/pipelines/[UUID][0]/updates",
           "body": {
             "development": true,
             "refresh_selection": [
        -      "table1",
        -      "table2"
        +      "table1 table2"
             ]
           }
         }
        @@ -44,17 +36,16 @@
         === Running pipeline with --full-refresh flag and specific tables
         
         >>> run_pipeline --full-refresh table1 table2
        -• Full refreshed [table1, table2]
        +• Full refreshed [table1 table2]
         
         >>> print_requests.py //pipelines
         {
           "method": "POST",
        -  "path": "/api/2.0/pipelines/[UUID]/updates",
        +  "path": "/api/2.0/pipelines/[UUID][0]/updates",
           "body": {
             "development": true,
             "full_refresh_selection": [
        -      "table1",
        -      "table2"
        +      "table1 table2"
             ]
           }
         }
        @@ -62,22 +46,20 @@
         === Running pipeline with --full-refresh flag and --refresh flag
         
         >>> run_pipeline --full-refresh table:table1 table:table2 --refresh table:table3 table:table4
        -• Refreshed [table3, table4]
        -• Full refreshed [table1, table2]
        +• Refreshed [table:table3 table:table4]
        +• Full refreshed [table:table1 table:table2]
         
         >>> print_requests.py //pipelines
         {
           "method": "POST",
        -  "path": "/api/2.0/pipelines/[UUID]/updates",
        +  "path": "/api/2.0/pipelines/[UUID][0]/updates",
           "body": {
             "development": true,
             "full_refresh_selection": [
        -      "table1",
        -      "table2"
        +      "table:table1 table:table2"
             ],
             "refresh_selection": [
        -      "table3",
        -      "table4"
        +      "table:table3 table:table4"
             ]
           }
         }
        @@ -85,21 +58,20 @@
         === Refreshing tables and fully refreshing a source file together
         
         >>> run_pipeline --refresh table:table3 table:table4 --full-refresh file:pipeline_file.py
        -• Refreshed [table3, table4]
        -• Full refreshed tables defined in [/Workspace/Users/[USERNAME]/.bundle/test-pipeline-run-flags/[UNIQUE_NAME]/files/pipeline_file.py]
        +• Refreshed [table:table3 table:table4]
        +• Full refreshed [file:pipeline_file.py]
         
         >>> print_requests.py //pipelines
         {
           "method": "POST",
        -  "path": "/api/2.0/pipelines/[UUID]/updates",
        +  "path": "/api/2.0/pipelines/[UUID][0]/updates",
           "body": {
             "development": true,
        -    "full_refresh_selection_by_path": [
        -      "/Workspace/Users/[USERNAME]/.bundle/test-pipeline-run-flags/[UNIQUE_NAME]/files/pipeline_file.py"
        +    "full_refresh_selection": [
        +      "file:pipeline_file.py"
             ],
             "refresh_selection": [
        -      "table3",
        -      "table4"
        +      "table:table3 table:table4"
             ]
           }
         }
        @@ -107,16 +71,16 @@
         === Refreshing tables defined in a source file
         
         >>> run_pipeline --refresh file:pipeline_file.py
        -• Refreshed tables defined in [/Workspace/Users/[USERNAME]/.bundle/test-pipeline-run-flags/[UNIQUE_NAME]/files/pipeline_file.py]
        +• Refreshed [file:pipeline_file.py]
         
         >>> print_requests.py //pipelines
         {
           "method": "POST",
        -  "path": "/api/2.0/pipelines/[UUID]/updates",
        +  "path": "/api/2.0/pipelines/[UUID][0]/updates",
           "body": {
             "development": true,
        -    "refresh_selection_by_path": [
        -      "/Workspace/Users/[USERNAME]/.bundle/test-pipeline-run-flags/[UNIQUE_NAME]/files/pipeline_file.py"
        +    "refresh_selection": [
        +      "file:pipeline_file.py"
             ]
           }
         }
        @@ -124,16 +82,16 @@
         === Fully refreshing tables defined in a source file
         
         >>> run_pipeline --full-refresh file:pipeline_file.py
        -• Full refreshed tables defined in [/Workspace/Users/[USERNAME]/.bundle/test-pipeline-run-flags/[UNIQUE_NAME]/files/pipeline_file.py]
        +• Full refreshed [file:pipeline_file.py]
         
         >>> print_requests.py //pipelines
         {
           "method": "POST",
        -  "path": "/api/2.0/pipelines/[UUID]/updates",
        +  "path": "/api/2.0/pipelines/[UUID][0]/updates",
           "body": {
             "development": true,
        -    "full_refresh_selection_by_path": [
        -      "/Workspace/Users/[USERNAME]/.bundle/test-pipeline-run-flags/[UNIQUE_NAME]/files/pipeline_file.py"
        +    "full_refresh_selection": [
        +      "file:pipeline_file.py"
             ]
           }
         }
        @@ -141,16 +93,16 @@
         === Refreshing a source notebook
         
         >>> run_pipeline --refresh file:src/pipeline_notebook.sql
        -• Refreshed tables defined in [/Workspace/Users/[USERNAME]/.bundle/test-pipeline-run-flags/[UNIQUE_NAME]/files/src/pipeline_notebook]
        +• Refreshed [file:src/pipeline_notebook.sql]
         
         >>> print_requests.py //pipelines
         {
           "method": "POST",
        -  "path": "/api/2.0/pipelines/[UUID]/updates",
        +  "path": "/api/2.0/pipelines/[UUID][0]/updates",
           "body": {
             "development": true,
        -    "refresh_selection_by_path": [
        -      "/Workspace/Users/[USERNAME]/.bundle/test-pipeline-run-flags/[UNIQUE_NAME]/files/src/pipeline_notebook"
        +    "refresh_selection": [
        +      "file:src/pipeline_notebook.sql"
             ]
           }
         }
        @@ -158,16 +104,16 @@
         === Refreshing an absolute workspace path
         
         >>> run_pipeline --refresh file:/Workspace/Users/[USERNAME]/.bundle/test-pipeline-run-flags/[UNIQUE_NAME]/files/pipeline_file.py
        -• Refreshed tables defined in [/Workspace/Users/[USERNAME]/.bundle/test-pipeline-run-flags/[UNIQUE_NAME]/files/pipeline_file.py]
        +• Refreshed [file:/Workspace/Users/[USERNAME]/.bundle/test-pipeline-run-flags/[UNIQUE_NAME]/files/pipeline_file.py]
         
         >>> print_requests.py //pipelines
         {
           "method": "POST",
        -  "path": "/api/2.0/pipelines/[UUID]/updates",
        +  "path": "/api/2.0/pipelines/[UUID][0]/updates",
           "body": {
             "development": true,
        -    "refresh_selection_by_path": [
        -      "/Workspace/Users/[USERNAME]/.bundle/test-pipeline-run-flags/[UNIQUE_NAME]/files/pipeline_file.py"
        +    "refresh_selection": [
        +      "file:/Workspace/Users/[USERNAME]/.bundle/test-pipeline-run-flags/[UNIQUE_NAME]/files/pipeline_file.py"
... (246 more lines truncated, raise DATABRICKS_CLI_TEST_MAX_LOG to see more)

@chenyuem-db
chenyuem-db force-pushed the chenyuem-db/pipeline-refresh-by-file branch from ae8800b to 9fd46d5 Compare October 28, 2026 22:30
@chenyuem-db
chenyuem-db marked this pull request as ready for review October 29, 2026 22:08
@chenyuem-db
chenyuem-db force-pushed the chenyuem-db/pipeline-refresh-by-file branch 2 times, most recently from e092a9c to 9d78b59 Compare November 1, 2026 22:53
Comment thread bundle/run/pipeline_options.go Resolved
Comment thread bundle/run/pipeline_options.go Resolved
Comment thread cmd/pipelines/run.go Resolved
Comment thread bundle/run/pipeline_refresh.go Resolved
Comment thread cmd/pipelines/refresh.go Resolved
Comment thread bundle/run/pipeline_refresh.go Resolved
Comment thread bundle/run/pipeline_refresh.go Resolved
Comment thread cmd/pipelines/refresh.go Resolved

@dhruvgoel92 dhruvgoel92 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

API changes lgtm.

Please get 1 more sign-off on code changes.

@schulte-lukas schulte-lukas left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI Review Requests Changes

  1. [P1] File selection with --restart can stop a continuous pipeline before the replacement is rejected.

    Pre-restart validation checks selector syntax and flag combinations, but does not check the pipeline's execution mode.

    For a continuous pipeline, a command such as --restart --refresh file:src/orders.py passes validation, stops the active update, and then sends a request the backend rejects. The backend explicitly rejects ordinary file selections in continuous mode. There are separate flag-controlled wildcard exceptions, but they do not make this file-only request valid.

    Required fix: validate supported selector/mode combinations against the deployed pipeline before cancellation. Add a regression test asserting that the rejected request sends no stop call.

  2. [P1, compatibility] Existing commands can silently select different full-refresh targets.

    The new parser changes more than comma-separated lists:

    databricks bundle run my_pipeline --full-refresh 'order details'

    Previously, this selected one table named order details. It now selects two tables, order and details. If those tables exist, the command can reset different tables without an error. I reproduced the changed payload.

    Comma-list removal is explicitly intentional, but the migration error only protects comma inputs. Existing shell-quoted names receive no such protection.

    Release blocker: preserve existing semantics, or introduce the new grammar through an explicit migration/versioning mechanism. A changelog entry alone does not protect automation from silent target changes.

  3. [P2] Publishing mode is inferred incorrectly, which corrupts quoted DPM identifiers.

    Payload construction assumes pipeline.Schema == "" means legacy publishing mode and removes SQL quoting.

    That assumption is false. DPM pipelines can use the deprecated target field while leaving schema unset. The backend mode helper explicitly documents this case.

    My probe showed a quoted identifier containing an escaped backtick being converted into an unquoted identifier with a single, unescaped backtick. That changes SQL identifier semantics and can cause parsing failures. The current "both publishing modes" tests encode the same incorrect schema-based assumption.

    Required fix: use the backend-reported effective publishing mode, and test DPM-with-target separately from true legacy pipelines.

  4. [P2] In-place deployments incorrectly require selected sources to belong to the upload set.

    The resolver requires every relative source to appear in GetSyncFiles(), including in-place/source-linked deployments.

    Those deployments reference existing Workspace sources directly and skip uploads. A pipeline source excluded by sync.exclude can therefore remain a valid deployed library, while --refresh file:orders.py rejects it as excluded from sync. I reproduced this rejection. Using its absolute /Workspace/... selector bypasses that check, creating inconsistent behavior for the same source.

    Required fix: resolve in-place sources using their actual source paths without imposing upload eligibility.

@andrewnester andrewnester left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nits:

  • The selector help block in cmd/bundle/run.go:111-124 and cmd/pipelines/run.go:242-256 is the same text twice, and only the pipelines copy says paths are relative to the current directory. Can it be one constant next to DefineRefreshFlags?
  • In both refresh-flags scripts the 2>&1 | grep -E '^(>>> |Error:)' filter drops the blank line trace prints, so output.txt shows === Rejecting invalid selector syntax before restart>>> musterr ... on one line.

if !ok {
if strings.HasPrefix(selector, "flow:") {
return nil, nil, errors.New("flow selectors are not supported yet; select tables or files")
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In in-place mode the sync root itself lives under /Workspace/, so file:/Workspace/Users/me/proj/src/nb.py takes this branch and is sent verbatim with its .py extension, while file:src/nb.py for the same file goes through the in-place branch below and gets the notebook extension stripped, which is the path the deployed library uses (translateNotebookPath in bundle/config/mutator/translate_paths.go). Should an absolute path that resolves inside SyncRootPath go through the local branch, with only paths outside it treated as remote?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Absolute source paths inside the in-place sync root now use the same notebook mapping as relative paths. Other Workspace paths are unchanged.


This comment was generated with GitHub MCP.

Comment thread bundle/run/pipeline.go Outdated

func (r *pipelineRunner) Restart(ctx context.Context, opts *Options) (output.RunOutput, error) {
hasFiles := len(opts.Pipeline.RefreshByPath) > 0 || len(opts.Pipeline.FullRefreshByPath) > 0
if hasFiles && !slices.Contains(opts.Pipeline.Refresh, "*") {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the "*" selector doing here? It isn't in the help text or anywhere else in the CLI, and with --refresh "*" --full-refresh file:x --restart on a continuous pipeline it skips this check, so the active update is stopped before the backend rejects the request. A short comment on why the check lives only in Restart (I assume the backend rejects the request either way, but Restart has already cancelled the active update by then) would also save the next reader the trip through Run.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed the * exception and updated the regression tests to confirm rejected requests never stop the active update.


This comment was generated with GitHub MCP.

Comment thread cmd/pipelines/run.go
}

events, err := fetchAllPipelineEvents(ctx, w, pipelineId, params)
// The public events API rejects event_type filters.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cmd/pipelines/logs.go still sends event_type in (...) to the same /events endpoint through fetchAllPipelineEvents (buildPipelineEventFilter), so if the public API rejects that filter, pipelines logs --event-type is broken the same way. Is the rejection specific to combining it with update_id, or does logs need the same fix? This also now pages through every event of the update, flow_progress included, to keep a handful of update_progress ones, where the old call fetched one page. Since this fix is independent of file selectors, can it be its own PR covering both call sites?

@chenyuem-db chenyuem-db Nov 11, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I’m leaving this unchanged for now since it works on some workspaces. I’ll try to address this in a follow-up PR.


This comment was generated with GitHub MCP.

Comment thread bundle/run/pipeline.go
if err != nil {
return nil, err
}
// Unset modes follow the backend's schema-based legacy detection.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This says how an unset mode is classified but not why the two modes get different payloads: legacy gets order details, default publishing mode gets `order details`. Can you add the API-side reason (does LPM take raw table names while DPM parses SQL identifiers?), and was the quoted form checked against a real DPM pipeline, i.e. did table1 actually refresh rather than the update completing with nothing selected? The acceptance output only echoes the request back.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added the API-side explanation. I also verified quoted --refresh and --full-refresh in staging: only the selected table’s data changed.


This comment was generated with GitHub MCP.

if !filepath.IsLocal(rel) {
return nil, nil, fmt.Errorf("source file %q is outside the bundle sync root", selector)
}
if b.IsSourceLinkedDeployment() {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This branch computes the remote name by hand (DetectWithFS + StripExtension) while the upload branch below gets it from sync.NewSnapshotState, which does the same two steps for every synced file. Can the name be computed one way for both, with the sync file list used only to reject excluded files in upload mode? Or is this not worth it?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I’ve kept upload mode on the sync snapshot to preserve its naming and collision checks; relative and absolute in-place paths now share the same helper.


This comment was generated with GitHub MCP.

Comment thread bundle/bundle.go Outdated
return list.Files(ctx)
}

// RemoteFilePath returns the initialized sync destination.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"the initialized sync destination" doesn't say what is decided or from what. Something like: "RemoteFilePath returns the workspace path deployed resources reference bundle files under: the sync root for in-place deployments, where files are not copied, otherwise workspace.file_path." bundle/deploy/metadata/compute.go:131-135 computes the same value inline; can it use this helper too?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Clarified what the helper returns and reused it in metadata computation.


This comment was generated with GitHub MCP.

// List of tables or source files to update.
Refresh []string

RefreshByPath []string

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

RefreshByPath and FullRefreshByPath have no doc comment, and Refresh is described as "tables or source files" although after ResolveRefreshSelectors it holds table names only, with table: stripped and backticks kept. Can the comments say which stage each field belongs to? Same for the ResolveRefreshSelectors doc comment, which mentions only files but also strips table:, rejects flow:, and runs Validate.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated the field comments to explain their values before and after resolution, and expanded the resolver comment to include validation.


This comment was generated with GitHub MCP.

@@ -0,0 +1 @@
* Support `file:<path>` selectors in `pipelines run` and pipeline `bundle run` with `--refresh` and `--full-refresh`. Both flags now accept space-separated selectors inside quotes or repeated flags. Replace comma-separated lists such as `--refresh orders,payments` with `--refresh "table:orders table:payments"`. Use backticks for names or paths containing spaces or commas, and double backticks to escape a literal backtick. ([#7184](https://github.com/databricks/cli/pull/7184))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

--refresh orders,payments worked before and now errors, so this breaks existing scripts. Should it go under notable-changes rather than cli? The entry could also be trimmed to the change and the migration; the backtick rules are already in the help text.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moved this to notable-changes and shortened it to the feature and migration examples.


This comment was generated with GitHub MCP.

idelafuente-oc pushed a commit to idelafuente-oc/cli that referenced this pull request Nov 11, 2026
…databricks#7386)

## Changes
Adds one bullet to pass 9 (Tests) of
`.agents/skills/pr-review/SKILL.md`, placed after the bullet on whether
each new or changed test shows the effect it claims. When a change sends
a new field, value format or API call, and no covering test is `Cloud =
true` and the description does not mention a real-workspace run, the
review asks whether it was verified against one. Nothing else in the
skill changes. No `.nextchanges/` fragment is added, matching databricks#7344,
which added the skill.

## Why
The fake server in `libs/testserver` only returns what its handlers were
taught. A test that records the request, or a handler that echoes a
field back, shows what the CLI sends, not what the API accepts. Pass 9
never asked whether such a change had been checked against a real
workspace, and maintainers ask that question often.

The gap showed up in a review of databricks#7184 ("Support file selectors in
pipelines run"). The change started sending new
`refresh_selection_by_path` values (workspace paths, notebook extension
stripped) to the pipelines StartUpdate API. The only coverage was the
fake server echoing them back. The test was not `Cloud = true`, and the
PR description said no live-workspace run was done. The first review
question was whether it had been run against a real workspace, and the
skill as written would not have raised it.

The bullet asks a question rather than requiring a cloud run. Every
acceptance test still runs locally, and `Cloud = true` only adds a run
against a real workspace.

## Tests
- Markdown-only change. No Go code or acceptance tests changed.
- Ran the whitespace validator that `./task ws` runs, without `--fix`:
10787 checked, 0 failed.
- Checked the bullet's claims against `.agents/rules/testing.md`: `Cloud
= true` adds a cloud run and never removes the local one, `Cloud` is
inherited (so `out.test.toml` holds the effective value), and the fake
server lives in `libs/testserver`.

This pull request and its description were written by Isaac.

Co-authored-by: Isaac <no-reply@databricks.com>
@chenyuem-db
chenyuem-db force-pushed the chenyuem-db/pipeline-refresh-by-file branch from 1397221 to c27cef9 Compare November 11, 2026 20:13
@chenyuem-db

Copy link
Copy Markdown
Contributor Author

@andrewnester, on the two nits:

  • Both commands now share one help constant, including the current-directory rule for local paths.
  • Rebased onto main’s title newline fix and regenerated the affected snapshots.

This comment was generated with GitHub MCP.

tedjt pushed a commit to tedjt/cli that referenced this pull request Nov 12, 2026
…abricks#7464)

## Changes
Pass 8 of the pr-review skill listed new doc comments with a Go-only
grep. This widens the pass to Python, shell and YAML (the languages
`.agents/rules/comments.md` covers), counts `acceptance/bin/*.py`
docstrings and `--help` text and the doc comment of a new test function
in test-only changes, and asks the reviewer to check the other callers
of an API when a new comment claims a quirk of it.

## Why
In reviews of databricks#7187, databricks#7462, databricks#6923, databricks#3518, databricks#6714 and databricks#7184 the pass
produced an empty table on exactly the changes where the comment was the
main thing to judge: a YAML labeler config, a Python-only comment block,
a test-only invariant test, a shared acceptance helper. The findings on
those PRs came from reading `comments.md` by hand instead. On databricks#7184 a
new comment asserted an API rejection that another caller of the same
endpoint contradicts.

## Tests
Documentation only, no code changes.

This pull request and its description were written by Isaac.

Co-authored-by: Isaac <no-reply@databricks.com>
@chenyuem-db
chenyuem-db added this pull request to the merge queue Nov 12, 2026
Merged via the queue into main with commit 58711eb Nov 12, 2026
30 checks passed
@chenyuem-db
chenyuem-db deleted the chenyuem-db/pipeline-refresh-by-file branch November 12, 2026 18:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ai-assisted DABs DABs related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants