Repository navigation
Support file selectors in pipelines run - #7184
Conversation
✅ Integration test reportCommit: 879c75e
Top 12 slowest tests (at least 2 minutes):
|
0824056 to
652cd9e
Compare
|
Commit: Regression test reportSelected 3 changed tests (limit=20, 0 not selected) 5 tests: regression=5 unreleased=0 coverage=0 failing=0
TestAccept/bundle/help/bundle-run/DMS=: branch ✅ | main (a0a6430) ❌ | latest (v1.26.0) ❌main (a0a6430): latest (v1.26.0): same output as main (a0a6430) TestAccept/bundle/help/bundle-run/DMS=true: branch ✅ | main (a0a6430) ❌ | latest (v1.26.0) ❌main (a0a6430): latest (v1.26.0): same output as main (a0a6430) TestAccept/bundle/run/refresh-flags/DMS=: branch ✅ | main (a0a6430) ❌ | latest (v1.26.0) ❌main (a0a6430): latest (v1.26.0): TestAccept/bundle/run/refresh-flags/DMS=true: branch ✅ | main (a0a6430) ❌ | latest (v1.26.0) ❌main (a0a6430): latest (v1.26.0): TestAccept/pipelines/run/refresh-flags: branch ✅ | main (a0a6430) ❌ | latest (v1.26.0) ❌main (a0a6430): latest (v1.26.0): |
ae8800b to
9fd46d5
Compare
e092a9c to
9d78b59
Compare
005ad20 to
ee50d88
Compare
dhruvgoel92
left a comment
There was a problem hiding this comment.
API changes lgtm.
Please get 1 more sign-off on code changes.
There was a problem hiding this comment.
AI Review Requests Changes
-
[P1] File selection with
--restartcan stop a continuous pipeline before the replacement is rejected.Pre-restart validation checks selector syntax and flag combinations, but does not check the pipeline's execution mode.
For a continuous pipeline, a command such as
--restart --refresh file:src/orders.pypasses validation, stops the active update, and then sends a request the backend rejects. The backend explicitly rejects ordinary file selections in continuous mode. There are separate flag-controlled wildcard exceptions, but they do not make this file-only request valid.Required fix: validate supported selector/mode combinations against the deployed pipeline before cancellation. Add a regression test asserting that the rejected request sends no stop call.
-
[P1, compatibility] Existing commands can silently select different full-refresh targets.
The new parser changes more than comma-separated lists:
databricks bundle run my_pipeline --full-refresh 'order details'Previously, this selected one table named
order details. It now selects two tables,orderanddetails. If those tables exist, the command can reset different tables without an error. I reproduced the changed payload.Comma-list removal is explicitly intentional, but the migration error only protects comma inputs. Existing shell-quoted names receive no such protection.
Release blocker: preserve existing semantics, or introduce the new grammar through an explicit migration/versioning mechanism. A changelog entry alone does not protect automation from silent target changes.
-
[P2] Publishing mode is inferred incorrectly, which corrupts quoted DPM identifiers.
Payload construction assumes
pipeline.Schema == ""means legacy publishing mode and removes SQL quoting.That assumption is false. DPM pipelines can use the deprecated
targetfield while leavingschemaunset. The backend mode helper explicitly documents this case.My probe showed a quoted identifier containing an escaped backtick being converted into an unquoted identifier with a single, unescaped backtick. That changes SQL identifier semantics and can cause parsing failures. The current "both publishing modes" tests encode the same incorrect schema-based assumption.
Required fix: use the backend-reported effective publishing mode, and test DPM-with-
targetseparately from true legacy pipelines. -
[P2] In-place deployments incorrectly require selected sources to belong to the upload set.
The resolver requires every relative source to appear in
GetSyncFiles(), including in-place/source-linked deployments.Those deployments reference existing Workspace sources directly and skip uploads. A pipeline source excluded by
sync.excludecan therefore remain a valid deployed library, while--refresh file:orders.pyrejects it as excluded from sync. I reproduced this rejection. Using its absolute/Workspace/...selector bypasses that check, creating inconsistent behavior for the same source.Required fix: resolve in-place sources using their actual source paths without imposing upload eligibility.
andrewnester
left a comment
There was a problem hiding this comment.
Nits:
- The selector help block in
cmd/bundle/run.go:111-124andcmd/pipelines/run.go:242-256is the same text twice, and only the pipelines copy says paths are relative to the current directory. Can it be one constant next toDefineRefreshFlags? - In both
refresh-flagsscripts the2>&1 | grep -E '^(>>> |Error:)'filter drops the blank linetraceprints, sooutput.txtshows=== Rejecting invalid selector syntax before restart>>> musterr ...on one line.
| if !ok { | ||
| if strings.HasPrefix(selector, "flow:") { | ||
| return nil, nil, errors.New("flow selectors are not supported yet; select tables or files") | ||
| } |
There was a problem hiding this comment.
In in-place mode the sync root itself lives under /Workspace/, so file:/Workspace/Users/me/proj/src/nb.py takes this branch and is sent verbatim with its .py extension, while file:src/nb.py for the same file goes through the in-place branch below and gets the notebook extension stripped, which is the path the deployed library uses (translateNotebookPath in bundle/config/mutator/translate_paths.go). Should an absolute path that resolves inside SyncRootPath go through the local branch, with only paths outside it treated as remote?
There was a problem hiding this comment.
Absolute source paths inside the in-place sync root now use the same notebook mapping as relative paths. Other Workspace paths are unchanged.
This comment was generated with GitHub MCP.
|
|
||
| func (r *pipelineRunner) Restart(ctx context.Context, opts *Options) (output.RunOutput, error) { | ||
| hasFiles := len(opts.Pipeline.RefreshByPath) > 0 || len(opts.Pipeline.FullRefreshByPath) > 0 | ||
| if hasFiles && !slices.Contains(opts.Pipeline.Refresh, "*") { |
There was a problem hiding this comment.
What is the "*" selector doing here? It isn't in the help text or anywhere else in the CLI, and with --refresh "*" --full-refresh file:x --restart on a continuous pipeline it skips this check, so the active update is stopped before the backend rejects the request. A short comment on why the check lives only in Restart (I assume the backend rejects the request either way, but Restart has already cancelled the active update by then) would also save the next reader the trip through Run.
There was a problem hiding this comment.
Removed the * exception and updated the regression tests to confirm rejected requests never stop the active update.
This comment was generated with GitHub MCP.
| } | ||
|
|
||
| events, err := fetchAllPipelineEvents(ctx, w, pipelineId, params) | ||
| // The public events API rejects event_type filters. |
There was a problem hiding this comment.
cmd/pipelines/logs.go still sends event_type in (...) to the same /events endpoint through fetchAllPipelineEvents (buildPipelineEventFilter), so if the public API rejects that filter, pipelines logs --event-type is broken the same way. Is the rejection specific to combining it with update_id, or does logs need the same fix? This also now pages through every event of the update, flow_progress included, to keep a handful of update_progress ones, where the old call fetched one page. Since this fix is independent of file selectors, can it be its own PR covering both call sites?
There was a problem hiding this comment.
I’m leaving this unchanged for now since it works on some workspaces. I’ll try to address this in a follow-up PR.
This comment was generated with GitHub MCP.
| if err != nil { | ||
| return nil, err | ||
| } | ||
| // Unset modes follow the backend's schema-based legacy detection. |
There was a problem hiding this comment.
This says how an unset mode is classified but not why the two modes get different payloads: legacy gets order details, default publishing mode gets `order details`. Can you add the API-side reason (does LPM take raw table names while DPM parses SQL identifiers?), and was the quoted form checked against a real DPM pipeline, i.e. did table1 actually refresh rather than the update completing with nothing selected? The acceptance output only echoes the request back.
There was a problem hiding this comment.
Added the API-side explanation. I also verified quoted --refresh and --full-refresh in staging: only the selected table’s data changed.
This comment was generated with GitHub MCP.
| if !filepath.IsLocal(rel) { | ||
| return nil, nil, fmt.Errorf("source file %q is outside the bundle sync root", selector) | ||
| } | ||
| if b.IsSourceLinkedDeployment() { |
There was a problem hiding this comment.
This branch computes the remote name by hand (DetectWithFS + StripExtension) while the upload branch below gets it from sync.NewSnapshotState, which does the same two steps for every synced file. Can the name be computed one way for both, with the sync file list used only to reject excluded files in upload mode? Or is this not worth it?
There was a problem hiding this comment.
I’ve kept upload mode on the sync snapshot to preserve its naming and collision checks; relative and absolute in-place paths now share the same helper.
This comment was generated with GitHub MCP.
| return list.Files(ctx) | ||
| } | ||
|
|
||
| // RemoteFilePath returns the initialized sync destination. |
There was a problem hiding this comment.
"the initialized sync destination" doesn't say what is decided or from what. Something like: "RemoteFilePath returns the workspace path deployed resources reference bundle files under: the sync root for in-place deployments, where files are not copied, otherwise workspace.file_path." bundle/deploy/metadata/compute.go:131-135 computes the same value inline; can it use this helper too?
There was a problem hiding this comment.
Clarified what the helper returns and reused it in metadata computation.
This comment was generated with GitHub MCP.
| // List of tables or source files to update. | ||
| Refresh []string | ||
|
|
||
| RefreshByPath []string |
There was a problem hiding this comment.
RefreshByPath and FullRefreshByPath have no doc comment, and Refresh is described as "tables or source files" although after ResolveRefreshSelectors it holds table names only, with table: stripped and backticks kept. Can the comments say which stage each field belongs to? Same for the ResolveRefreshSelectors doc comment, which mentions only files but also strips table:, rejects flow:, and runs Validate.
There was a problem hiding this comment.
Updated the field comments to explain their values before and after resolution, and expanded the resolver comment to include validation.
This comment was generated with GitHub MCP.
| @@ -0,0 +1 @@ | |||
| * Support `file:<path>` selectors in `pipelines run` and pipeline `bundle run` with `--refresh` and `--full-refresh`. Both flags now accept space-separated selectors inside quotes or repeated flags. Replace comma-separated lists such as `--refresh orders,payments` with `--refresh "table:orders table:payments"`. Use backticks for names or paths containing spaces or commas, and double backticks to escape a literal backtick. ([#7184](https://github.com/databricks/cli/pull/7184)) | |||
There was a problem hiding this comment.
--refresh orders,payments worked before and now errors, so this breaks existing scripts. Should it go under notable-changes rather than cli? The entry could also be trimmed to the change and the migration; the backtick rules are already in the help text.
There was a problem hiding this comment.
Moved this to notable-changes and shortened it to the feature and migration examples.
This comment was generated with GitHub MCP.
…databricks#7386) ## Changes Adds one bullet to pass 9 (Tests) of `.agents/skills/pr-review/SKILL.md`, placed after the bullet on whether each new or changed test shows the effect it claims. When a change sends a new field, value format or API call, and no covering test is `Cloud = true` and the description does not mention a real-workspace run, the review asks whether it was verified against one. Nothing else in the skill changes. No `.nextchanges/` fragment is added, matching databricks#7344, which added the skill. ## Why The fake server in `libs/testserver` only returns what its handlers were taught. A test that records the request, or a handler that echoes a field back, shows what the CLI sends, not what the API accepts. Pass 9 never asked whether such a change had been checked against a real workspace, and maintainers ask that question often. The gap showed up in a review of databricks#7184 ("Support file selectors in pipelines run"). The change started sending new `refresh_selection_by_path` values (workspace paths, notebook extension stripped) to the pipelines StartUpdate API. The only coverage was the fake server echoing them back. The test was not `Cloud = true`, and the PR description said no live-workspace run was done. The first review question was whether it had been run against a real workspace, and the skill as written would not have raised it. The bullet asks a question rather than requiring a cloud run. Every acceptance test still runs locally, and `Cloud = true` only adds a run against a real workspace. ## Tests - Markdown-only change. No Go code or acceptance tests changed. - Ran the whitespace validator that `./task ws` runs, without `--fix`: 10787 checked, 0 failed. - Checked the bullet's claims against `.agents/rules/testing.md`: `Cloud = true` adds a cloud run and never removes the local one, `Cloud` is inherited (so `out.test.toml` holds the effective value), and the fake server lives in `libs/testserver`. This pull request and its description were written by Isaac. Co-authored-by: Isaac <no-reply@databricks.com>
1397221 to
c27cef9
Compare
|
@andrewnester, on the two nits:
This comment was generated with GitHub MCP. |
…abricks#7464) ## Changes Pass 8 of the pr-review skill listed new doc comments with a Go-only grep. This widens the pass to Python, shell and YAML (the languages `.agents/rules/comments.md` covers), counts `acceptance/bin/*.py` docstrings and `--help` text and the doc comment of a new test function in test-only changes, and asks the reviewer to check the other callers of an API when a new comment claims a quirk of it. ## Why In reviews of databricks#7187, databricks#7462, databricks#6923, databricks#3518, databricks#6714 and databricks#7184 the pass produced an empty table on exactly the changes where the comment was the main thing to judge: a YAML labeler config, a Python-only comment block, a test-only invariant test, a shared acceptance helper. The findings on those PRs came from reading `comments.md` by hand instead. On databricks#7184 a new comment asserted an API rejection that another caller of the same endpoint contradicts. ## Tests Documentation only, no code changes. This pull request and its description were written by Isaac. Co-authored-by: Isaac <no-reply@databricks.com>
Changes
Add file-based refresh to
pipelines runand pipelinebundle run, bringing the UI's “Run file” workflow to the CLI.--refreshand--full-refreshaccept tables or source files.ordersortable:orders. Files requirefile:./Workspace/paths also work.--restartstops the active update.Deploy local edits before running. Each refresh mode accepts either tables or files. The two modes can use different types.
Why
Support SDP Turbo CLI project:
file:to avoid guessing from the local filesystem.Migration: replace
--refresh orders,paymentswith--refresh "orders payments", or repeat the flag. The same applies to--full-refresh. Old comma lists return an error with migration guidance.Showing resolved table names is a follow-up.
Tests