Skip to content

Wait for and print the results of databricks pipelines test - #7298

Open
tharunbhuk-db wants to merge 1 commit into
tharun/pipelines-test-print-resultsfrom
tharun/pipelines-test-pr2-output
Open

tharunbhuk-db wants to merge 1 commit into
tharun/pipelines-test-print-resultsfrom
tharun/pipelines-test-pr2-output

Conversation

@tharunbhuk-db

@tharunbhuk-db tharunbhuk-db commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

🥞 Stacked PR


  • Wait for the test update, print the results, and exit 0 only when at least one test ran and none failed or errored.
  • --no-wait returns after the update starts.
  • The wait follows the bundle pipeline runner's loop.

Tests: unit tests; acceptance in acceptance/pipelines/test/results/*, with the service stubbed per case.

This pull request and its description were written by Isaac.

@github-actions

github-actions Bot commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

Commit: d0d713865 | Workflow run

Regression test report

Selected 9 changed tests (limit=20, 0 not selected)

9 tests: regression=9 unreleased=0 coverage=0 failing=0
test branch main (688c228) latest (v1.25.0)
TestAccept/pipelines/databricks-cli-help ✅ ❌ ❌
TestAccept/pipelines/test/arguments ✅ ❌ ❌
TestAccept/pipelines/test/immutable ✅ ❌ ❌
TestAccept/pipelines/test/pipeline-id ✅ ❌ ❌
TestAccept/pipelines/test/results/failed ✅ ❌ ❌
TestAccept/pipelines/test/results/incomplete ✅ ❌ ❌
TestAccept/pipelines/test/results/passed ✅ ❌ ❌
TestAccept/pipelines/test/selection ✅ ❌ ❌
TestAccept/pipelines/test/submit-error ✅ ❌ ❌
TestAccept/pipelines/databricks-cli-help: branch ✅ | main (688c228) ❌ | latest (v1.25.0) ❌

main (688c228):

acceptance_test.go:1345: Diff:
        --- pipelines/databricks-cli-help/output.txt
        +++ /tmp/TestAcceptpipelinesdatabricks-cli-help4083411662/002/output.txt
        @@ -30,7 +30,6 @@
           run                   Run a pipeline
           show                  Preview a table's columns and sample rows
           stop                  Stop a pipeline
        -  test                  *Preview* Run the tests of a pipeline
         
         Management Commands
           apply-environment     *Public Preview* Apply the latest environment to the pipeline.

latest (v1.25.0): same output as main (688c228)

TestAccept/pipelines/test/arguments: branch ✅ | main (688c228) ❌ | latest (v1.25.0) ❌

main (688c228):

acceptance_test.go:1345: Diff:
        --- pipelines/test/arguments/output.txt
        +++ /tmp/TestAcceptpipelinestestarguments2531479417/002/output.txt
        @@ -1,73 +1,434 @@
         
         >>> [CLI] pipelines test --help
        -Run the tests of a pipeline in a test-only update.
        +The Lakeflow Spark Declarative Pipelines API allows you to create, edit,
        +  delete, start, and view details about pipelines.
         
        -This command is experimental. Set DATABRICKS_CLI_PIPELINES_TEST=true to enable it.
        +  Spark Declarative Pipelines is a framework for building reliable,
        +  maintainable, and testable data processing pipelines. You define the
        +  transformations to perform on your data, and Spark Declarative Pipelines
        +  manages task orchestration, cluster management, monitoring, data quality, and
        +  error handling.
         
        -KEY is the unique name of the pipeline, as defined in its YAML file.
        -It is required when test selections are given. If there is only one pipeline
        -in the project and no arguments are given, it is selected automatically.
        -Outside a project, identify the pipeline with --pipeline-id; all positional
        -arguments are then test selections.
        +  Instead of defining your data pipelines using a series of separate Apache
        +  Spark tasks, Spark Declarative Pipelines manages how your data is transformed
        +  based on a target schema you define for each processing step. You can also
        +  enforce data quality with Spark Declarative Pipelines expectations.
        +  Expectations allow you to define expected data quality and specify how to
        +  handle records that fail those expectations.
         
        -TEST_SELECTION is a test directory, file, or pytest test ID, for example
        -tests/test_orders.py::test_backfill. In a project, a relative path is a local
        -path, relative to the working directory, and is converted to its deployed
        -workspace path. With --pipeline-id, a relative path is relative to the
        -pipeline's root_path. Paths that start with / are workspace paths. Without a
        -selection, all of the pipeline's tests run.
        +Usage:
        +  databricks pipelines [flags]
        +  databricks pipelines [command]
         
        -Each --pytest-arg passes one argument to pytest, in the order given. Only -k
        -and -m are supported, each followed by its value in the next --pytest-arg, for
        -example:
        +Main Commands
        +  deploy                Deploy pipelines
        +  destroy               Destroy a pipelines project
        +  dry-run               Validate correctness of the pipeline's graph
        +  generate              Generate pipeline configuration
        +  history               Retrieve past runs for a pipeline
        +  init                  Initialize a new pipelines project
        +  logs                  Retrieve events for a pipeline
        +  open                  Open a pipeline in the browser
        +  run                   Run a pipeline
        +  show                  Preview a table's columns and sample rows
        +  stop                  Stop a pipeline
         
        -  databricks pipelines test my_pipeline --pytest-arg=-k --pytest-arg='orders and not slow'
        +Management Commands
        +  apply-environment     *Public Preview* Apply the latest environment to the pipeline.
        +  clone                 Clone a pipeline.
        +  create                Create a pipeline.
        +  delete                Delete a pipeline.
        +  get                   Get a pipeline.
        +  get-update            Get a pipeline update.
        +  list-pipeline-events  List pipeline events.
        +  list-pipelines        List pipelines.
        +  list-updates          List pipeline updates.
        +  start-update          Start a pipeline.
        +  update                Edit a pipeline.
         
        -The command waits for the results. It exits with code 0 only if at least one
        -test was collected and none failed or errored, so a run whose tests were all
        -skipped succeeds; otherwise it exits with code 1. --no-wait returns after the
        -update starts. To stop the update, run 'pipelines stop'.
        +Permission Commands
        +  get-permission-levels Get pipeline permission levels.
        +  get-permissions       Get pipeline permissions.
        +  set-permissions       Set pipeline permissions.
        +  update-permissions    Update pipeline permissions.
         
        +Flags:
        +  -h, --help          help for pipelines
        +      --var strings   set values for variables defined in project config. Example: --var="foo=bar"
        +
        +Global Flags:
        +      --debug            enable debug logging
        +  -o, --output type      output type: text or json (default text)
        +  -p, --profile string   ~/.databrickscfg profile
        +  -t, --target string    bundle target to use (if applicable)
        +
        +Use "databricks pipelines [command] --help" for more information about a command.
        +
        +=== The command is experimental and off by default
        +>>> musterr [CLI] pipelines test --pipeline-id 1234
        +Error: unknown flag: --pipeline-id
        +
         Usage:
        -  databricks pipelines test [flags] [KEY] [TEST_SELECTION...]
        +  databricks pipelines [flags]
        +  databricks pipelines [command]
         
        -Pipeline Test Flags:
        -      --no-wait                  Don't wait for the tests to complete.
        -      --pipeline-id string       ID of the pipeline to test, instead of a KEY.
        -      --pytest-arg stringArray   Argument to pass to pytest (repeatable). Only -k and -m are supported.
        +Main Commands
        +  deploy                Deploy pipelines
        +  destroy               Destroy a pipelines project
        +  dry-run               Validate correctness of the pipeline's graph
        +  generate              Generate pipeline configuration
        +  history               Retrieve past runs for a pipeline
        +  init                  Initialize a new pipelines project
        +  logs                  Retrieve events for a pipeline
        +  open                  Open a pipeline in the browser
        +  run                   Run a pipeline
        +  show                  Preview a table's columns and sample rows
        +  stop                  Stop a pipeline
         
        +Management Commands
        +  apply-environment     *Public Preview* Apply the latest environment to the pipeline.
        +  clone                 Clone a pipeline.
        +  create                Create a pipeline.
        +  delete                Delete a pipeline.
        +  get                   Get a pipeline.
        +  get-update            Get a pipeline update.
        +  list-pipeline-events  List pipeline events.
        +  list-pipelines        List pipelines.
        +  list-updates          List pipeline updates.
        +  start-update          Start a pipeline.
        +  update                Edit a pipeline.
        +
        +Permission Commands
        +  get-permission-levels Get pipeline permission levels.
        +  get-permissions       Get pipeline permissions.
        +  set-permissions       Set pipeline permissions.
        +  update-permissions    Update pipeline permissions.
        +
         Flags:
        -  -h, --help   help for test
        +  -h, --help          help for pipelines
        +      --var strings   set values for variables defined in project config. Example: --var="foo=bar"
         
         Global Flags:
               --debug            enable debug logging
           -o, --output type      output type: text or json (default text)
           -p, --profile string   ~/.databrickscfg profile
           -t, --target string    bundle target to use (if applicable)
        -      --var strings      set values for variables defined in project config. Example: --var="foo=bar"
         
        -=== The command is experimental and off by default
        ->>> musterr [CLI] pipelines test --pipeline-id 1234
        -Error: pipelines test is experimental. Set DATABRICKS_CLI_PIPELINES_TEST=true to enable it
        +Use "databricks pipelines [command] --help" for more information about a command.
         
        +
         === pytest arguments must be passed with --pytest-arg
         >>> musterr [CLI] pipelines test --pipeline-id 1234 -k orders
        -Error: -k is a pytest argument: pass pytest arguments with --pytest-arg, for example --pytest-arg=-k --pytest-arg='orders and not slow'
        +Error: unknown flag: --pipeline-id
         
        +Usage:
        +  databricks pipelines [flags]
        +  databricks pipelines [command]
        +
        +Main Commands
        +  deploy                Deploy pipelines
        +  destroy               Destroy a pipelines project
        +  dry-run               Validate correctness of the pipeline's graph
        +  generate              Generate pipeline configuration
        +  history               Retrieve past runs for a pipeline
        +  init                  Initialize a new pipelines project
        +  logs                  Retrieve events for a pipeline
        +  open                  Open a pipeline in the browser
        +  run                   Run a pipeline
        +  show                  Preview a table's columns and sample rows
        +  stop                  Stop a pipeline
        +
        +Management Commands
        +  apply-environment     *Public Preview* Apply the latest environment to the pipeline.
        +  clone                 Clone a pipeline.
        +  create                Create a pipeline.
        +  delete                Delete a pipeline.
        +  get                   Get a pipeline.
        +  get-update            Get a pipeline update.
        +  list-pipeline-events  List pipeline events.
        +  list-pipelines        List pipelines.
        +  list-updates          List pipeline updates.
        +  start-update          Start a pipeline.
        +  update                Edit a pipeline.
        +
        +Permission Commands
        +  get-permission-levels Get pipeline permission levels.
        +  get-permissions       Get pipeline permissions.
        +  set-permissions       Set pipeline permissions.
        +  update-permissions    Update pipeline permissions.
        +
        +Flags:
        +  -h, --help          help for pipelines
        +      --var strings   set values for variables defined in project config. Example: --var="foo=bar"
        +
        +Global Flags:
... (277 more lines truncated, raise DATABRICKS_CLI_TEST_MAX_LOG to see more)

latest (v1.25.0): same output as main (688c228)

TestAccept/pipelines/test/immutable: branch ✅ | main (688c228) ❌ | latest (v1.25.0) ❌

main (688c228):

acceptance_test.go:1345: Diff:
        --- pipelines/test/immutable/output.txt
        +++ /tmp/TestAcceptpipelinestestimmutable99414259/002/output.txt
        @@ -4,50 +4,112 @@
         Created pipelines.my_pipeline
         Files: 0 uploaded, 0 deleted
         Resources: 2 created, 0 changed, 0 deleted, 0 unchanged
        -View your pipeline my_pipeline here: [DATABRICKS_URL]/pipelines/[UUID][0]?w=[WORKSPACE_ID]
        +View your pipeline my_pipeline here: [DATABRICKS_URL]/pipelines/[UUID]?w=[WORKSPACE_ID]
         
         === Local test paths are not supported with workspace.sync_mode immutable
         >>> musterr [CLI] pipelines test my_pipeline tests/test_orders.py::test_backfill
        -Recommendation: This command runs the last deployed version of the code
        +Error: unknown command "test" for "databricks pipelines"
         
        -If you've made local changes, run 'databricks pipelines deploy' first to ensure they are included.
        +Usage:
        +  databricks pipelines [flags]
        +  databricks pipelines [command]
         
        -Error: cannot resolve test selection "tests/test_orders.py::test_backfill": local test paths are not supported with workspace.sync_mode immutable; pass an absolute workspace path
        +Main Commands
        +  deploy                Deploy pipelines
        +  destroy               Destroy a pipelines project
        +  dry-run               Validate correctness of the pipeline's graph
        +  generate              Generate pipeline configuration
        +  history               Retrieve past runs for a pipeline
        +  init                  Initialize a new pipelines project
        +  logs                  Retrieve events for a pipeline
        +  open                  Open a pipeline in the browser
        +  run                   Run a pipeline
        +  show                  Preview a table's columns and sample rows
        +  stop                  Stop a pipeline
         
        +Management Commands
        +  apply-environment     *Public Preview* Apply the latest environment to the pipeline.
        +  clone                 Clone a pipeline.
        +  create                Create a pipeline.
        +  delete                Delete a pipeline.
        +  get                   Get a pipeline.
        +  get-update            Get a pipeline update.
        +  list-pipeline-events  List pipeline events.
        +  list-pipelines        List pipelines.
        +  list-updates          List pipeline updates.
        +  start-update          Start a pipeline.
        +  update                Edit a pipeline.
        +
        +Permission Commands
        +  get-permission-levels Get pipeline permission levels.
        +  get-permissions       Get pipeline permissions.
        +  set-permissions       Set pipeline permissions.
        +  update-permissions    Update pipeline permissions.
        +
        +Flags:
        +  -h, --help          help for pipelines
        +      --var strings   set values for variables defined in project config. Example: --var="foo=bar"
        +
        +Global Flags:
        +      --debug            enable debug logging
        +  -o, --output type      output type: text or json (default text)
        +  -p, --profile string   ~/.databrickscfg profile
        +  -t, --target string    bundle target to use (if applicable)
        +
        +Use "databricks pipelines [command] --help" for more information about a command.
        +
        +
         === Absolute workspace paths and running all tests still work
         >>> MSYS_NO_PATHCONV=1 [CLI] pipelines test my_pipeline /Workspace/Users/someone@example.com/tests/test_orders.py::test_backfill --no-wait
        -Recommendation: This command runs the last deployed version of the code
        +Error: unknown flag: --no-wait
         
        -If you've made local changes, run 'databricks pipelines deploy' first to ensure they are included.
        +Usage:
        +  databricks pipelines [flags]
        +  databricks pipelines [command]
         
        -Pipeline: my_pipeline ([UUID][0])
        -Update: [UUID][1]
        +Main Commands
        +  deploy                Deploy pipelines
        +  destroy               Destroy a pipelines project
        +  dry-run               Validate correctness of the pipeline's graph
        +  generate              Generate pipeline configuration
        +  history               Retrieve past runs for a pipeline
        +  init                  Initialize a new pipelines project
        +  logs                  Retrieve events for a pipeline
        +  open                  Open a pipeline in the browser
        +  run                   Run a pipeline
        +  show                  Preview a table's columns and sample rows
        +  stop                  Stop a pipeline
         
        ->>> [CLI] pipelines test my_pipeline --no-wait
        -Recommendation: This command runs the last deployed version of the code
        +Management Commands
        +  apply-environment     *Public Preview* Apply the latest environment to the pipeline.
        +  clone                 Clone a pipeline.
        +  create                Create a pipeline.
        +  delete                Delete a pipeline.
        +  get                   Get a pipeline.
        +  get-update            Get a pipeline update.
        +  list-pipeline-events  List pipeline events.
        +  list-pipelines        List pipelines.
        +  list-updates          List pipeline updates.
        +  start-update          Start a pipeline.
        +  update                Edit a pipeline.
         
        -If you've made local changes, run 'databricks pipelines deploy' first to ensure they are included.
        +Permission Commands
        +  get-permission-levels Get pipeline permission levels.
        +  get-permissions       Get pipeline permissions.
        +  set-permissions       Set pipeline permissions.
        +  update-permissions    Update pipeline permissions.
         
        -Pipeline: my_pipeline ([UUID][0])
        -Update: [UUID][2]
        +Flags:
        +  -h, --help          help for pipelines
        +      --var strings   set values for variables defined in project config. Example: --var="foo=bar"
         
        ->>> print_requests.py //updates
        -{
        -  "method": "POST",
        -  "path": "/api/2.0/pipelines/[UUID][0]/updates",
        -  "body": {
        -    "test_only": true,
        -    "test_options": {
        -      "test_selection": [
        -        "/Workspace/Users/someone@example.com/tests/test_orders.py::test_backfill"
        -      ]
        -    }
        -  }
        -}
        -{
        -  "method": "POST",
        -  "path": "/api/2.0/pipelines/[UUID][0]/updates",
        -  "body": {
        -    "test_only": true
        -  }
        -}
        +Global Flags:
        +      --debug            enable debug logging
        +  -o, --output type      output type: text or json (default text)
        +  -p, --profile string   ~/.databrickscfg profile
        +  -t, --target string    bundle target to use (if applicable)
        +
        +Use "databricks pipelines [command] --help" for more information about a command.
        +
        +
        +Exit code: 1

latest (v1.25.0): same output as main (688c228)

TestAccept/pipelines/test/pipeline-id: branch ✅ | main (688c228) ❌ | latest (v1.25.0) ❌

main (688c228):

acceptance_test.go:1345: Diff:
        --- pipelines/test/pipeline-id/output.txt
        +++ /tmp/TestAcceptpipelinestestpipeline-id2780467119/002/output.txt
        @@ -1,50 +1,55 @@
         
         === Relative test selections resolve against the pipeline root_path
        ->>> MSYS_NO_PATHCONV=1 [CLI] pipelines test --pipeline-id [UUID][0] tests/test_orders.py::test_backfill /Workspace/Users/someone@example.com/other/tests --no-wait
        -Pipeline: [UUID][0]
        -Update: [UUID][1]
        +>>> MSYS_NO_PATHCONV=1 [CLI] pipelines test --pipeline-id [UUID] tests/test_orders.py::test_backfill /Workspace/Users/someone@example.com/other/tests --no-wait
        +Error: unknown flag: --pipeline-id
         
        ->>> print_requests.py //updates
        -{
        -  "method": "POST",
        -  "path": "/api/2.0/pipelines/[UUID][0]/updates",
        -  "body": {
        -    "test_only": true,
        -    "test_options": {
        -      "test_selection": [
        -        "/Workspace/Users/someone@example.com/orders/tests/test_orders.py::test_backfill",
        -        "/Workspace/Users/someone@example.com/other/tests"
        -      ]
        -    }
        -  }
        -}
        +Usage:
        +  databricks pipelines [flags]
        +  databricks pipelines [command]
         
        -=== Without a root_path, test selections must be workspace paths
        ->>> musterr [CLI] pipelines test --pipeline-id [UUID][2] tests
        -Error: cannot resolve test selection "tests": the pipeline has no root_path; use an absolute workspace path such as /Workspace/Users/someone@example.com/tests
        +Main Commands
        +  deploy                Deploy pipelines
        +  destroy               Destroy a pipelines project
        +  dry-run               Validate correctness of the pipeline's graph
        +  generate              Generate pipeline configuration
        +  history               Retrieve past runs for a pipeline
        +  init                  Initialize a new pipelines project
        +  logs                  Retrieve events for a pipeline
        +  open                  Open a pipeline in the browser
        +  run                   Run a pipeline
        +  show                  Preview a table's columns and sample rows
        +  stop                  Stop a pipeline
         
        ->>> MSYS_NO_PATHCONV=1 [CLI] pipelines test --pipeline-id [UUID][2] /Workspace/Users/someone@example.com/tests/test_other.py::test_x --no-wait
        -Pipeline: [UUID][2]
        -Update: [UUID][3]
        +Management Commands
        +  apply-environment     *Public Preview* Apply the latest environment to the pipeline.
        +  clone                 Clone a pipeline.
        +  create                Create a pipeline.
        +  delete                Delete a pipeline.
        +  get                   Get a pipeline.
        +  get-update            Get a pipeline update.
        +  list-pipeline-events  List pipeline events.
        +  list-pipelines        List pipelines.
        +  list-updates          List pipeline updates.
        +  start-update          Start a pipeline.
        +  update                Edit a pipeline.
         
        ->>> print_requests.py //updates
        -{
        -  "method": "POST",
        -  "path": "/api/2.0/pipelines/[UUID][2]/updates",
        -  "body": {
        -    "test_only": true,
        -    "test_options": {
        -      "test_selection": [
        -        "/Workspace/Users/someone@example.com/tests/test_other.py::test_x"
        -      ]
        -    }
        -  }
        -}
        +Permission Commands
        +  get-permission-levels Get pipeline permission levels.
        +  get-permissions       Get pipeline permissions.
        +  set-permissions       Set pipeline permissions.
        +  update-permissions    Update pipeline permissions.
         
        -=== Outside a project, --pipeline-id is required
        ->>> musterr [CLI] pipelines test tests
        -Error: Unable to locate the bundle root: databricks.yml not found.
        +Flags:
        +  -h, --help          help for pipelines
        +      --var strings   set values for variables defined in project config. Example: --var="foo=bar"
         
        -Run this command from a directory that contains a bundle, or set the DATABRICKS_BUNDLE_ROOT environment variable to the bundle root directory.
        -Alternatively, create a new bundle with 'databricks bundle init'.
        +Global Flags:
        +      --debug            enable debug logging
        +  -o, --output type      output type: text or json (default text)
        +  -p, --profile string   ~/.databrickscfg profile
        +  -t, --target string    bundle target to use (if applicable)
         
        +Use "databricks pipelines [command] --help" for more information about a command.
        +
        +
        +Exit code: 1

latest (v1.25.0): same output as main (688c228)

TestAccept/pipelines/test/results/failed: branch ✅ | main (688c228) ❌ | latest (v1.25.0) ❌

main (688c228):

acceptance_test.go:1345: Diff:
        --- pipelines/test/results/failed/output.txt
        +++ /tmp/TestAcceptpipelinestestresultsfailed1911077284/002/output.txt
        @@ -1,20 +1,53 @@
         
         === Failed, errored and skipped tests
        ->>> musterr [CLI] pipelines test --pipeline-id [UUID][0]
        -Pipeline: [UUID][0]
        -Update: [UUID][1]
        -Running tests...
        -1 of 4 PASSED tests/test_orders.py::test_valid_order ........................... [PASSED]
        -2 of 4 FAILED tests/test_orders.py::test_backfill .............................. [FAILED in 2.14s]
        -3 of 4 SKIPPED tests/test_orders.py::test_legacy_format ........................ [SKIPPED]
        -4 of 4 ERROR tests/test_orders.py::test_fixture ................................ [ERROR]
        +>>> musterr [CLI] pipelines test --pipeline-id [UUID]
        +Error: unknown flag: --pipeline-id
         
        -Completed with 1 failed and 1 errored tests:
        +Usage:
        +  databricks pipelines [flags]
        +  databricks pipelines [command]
         
        -Failure in tests/test_orders.py::test_backfill
        -  AssertionError: assert 100 == 150
        +Main Commands
        +  deploy                Deploy pipelines
        +  destroy               Destroy a pipelines project
        +  dry-run               Validate correctness of the pipeline's graph
        +  generate              Generate pipeline configuration
        +  history               Retrieve past runs for a pipeline
        +  init                  Initialize a new pipelines project
        +  logs                  Retrieve events for a pipeline
        +  open                  Open a pipeline in the browser
        +  run                   Run a pipeline
        +  show                  Preview a table's columns and sample rows
        +  stop                  Stop a pipeline
         
        -Error in tests/test_orders.py::test_fixture
        -  RuntimeError: fixture orders setup failed
        +Management Commands
        +  apply-environment     *Public Preview* Apply the latest environment to the pipeline.
        +  clone                 Clone a pipeline.
        +  create                Create a pipeline.
        +  delete                Delete a pipeline.
        +  get                   Get a pipeline.
        +  get-update            Get a pipeline update.
        +  list-pipeline-events  List pipeline events.
        +  list-pipelines        List pipelines.
        +  list-updates          List pipeline updates.
        +  start-update          Start a pipeline.
        +  update                Edit a pipeline.
         
        -Done. PASSED=1 FAILED=1 SKIPPED=1 ERROR=1 TOTAL=4
        +Permission Commands
        +  get-permission-levels Get pipeline permission levels.
        +  get-permissions       Get pipeline permissions.
        +  set-permissions       Set pipeline permissions.
        +  update-permissions    Update pipeline permissions.
        +
        +Flags:
        +  -h, --help          help for pipelines
        +      --var strings   set values for variables defined in project config. Example: --var="foo=bar"
        +
        +Global Flags:
        +      --debug            enable debug logging
        +  -o, --output type      output type: text or json (default text)
        +  -p, --profile string   ~/.databrickscfg profile
        +  -t, --target string    bundle target to use (if applicable)
        +
        +Use "databricks pipelines [command] --help" for more information about a command.
        +

latest (v1.25.0): same output as main (688c228)

TestAccept/pipelines/test/results/incomplete: branch ✅ | main (688c228) ❌ | latest (v1.25.0) ❌

main (688c228):

acceptance_test.go:1345: Diff:
        --- pipelines/test/results/incomplete/output.txt
        +++ /tmp/TestAcceptpipelinestestresultsincomplete4240715550/002/output.txt
        @@ -1,11 +1,53 @@
         
         === Compute ends before all results are received
        ->>> musterr [CLI] pipelines test --pipeline-id [UUID][0]
        -Pipeline: [UUID][0]
        -Update: [UUID][1]
        -Running tests...
        -1 of 2 PASSED tests/test_compute.py::test_small_order .......................... [PASSED]
        +>>> musterr [CLI] pipelines test --pipeline-id [UUID]
        +Error: unknown flag: --pipeline-id
         
        -RUN INCOMPLETE: the update failed before all test results were received.
        -Done. PASSED=1 FAILED=0 SKIPPED=0 ERROR=0 TOTAL=2
        -Error: update [UUID][1] is FAILED: ComputeTerminatedException: Compute terminated unexpectedly.
        +Usage:
        +  databricks pipelines [flags]
        +  databricks pipelines [command]
        +
        +Main Commands
        +  deploy                Deploy pipelines
        +  destroy               Destroy a pipelines project
        +  dry-run               Validate correctness of the pipeline's graph
        +  generate              Generate pipeline configuration
        +  history               Retrieve past runs for a pipeline
        +  init                  Initialize a new pipelines project
        +  logs                  Retrieve events for a pipeline
        +  open                  Open a pipeline in the browser
        +  run                   Run a pipeline
        +  show                  Preview a table's columns and sample rows
        +  stop                  Stop a pipeline
        +
        +Management Commands
        +  apply-environment     *Public Preview* Apply the latest environment to the pipeline.
        +  clone                 Clone a pipeline.
        +  create                Create a pipeline.
        +  delete                Delete a pipeline.
        +  get                   Get a pipeline.
        +  get-update            Get a pipeline update.
        +  list-pipeline-events  List pipeline events.
        +  list-pipelines        List pipelines.
        +  list-updates          List pipeline updates.
        +  start-update          Start a pipeline.
        +  update                Edit a pipeline.
        +
        +Permission Commands
        +  get-permission-levels Get pipeline permission levels.
        +  get-permissions       Get pipeline permissions.
        +  set-permissions       Set pipeline permissions.
        +  update-permissions    Update pipeline permissions.
        +
        +Flags:
        +  -h, --help          help for pipelines
        +      --var strings   set values for variables defined in project config. Example: --var="foo=bar"
        +
        +Global Flags:
        +      --debug            enable debug logging
        +  -o, --output type      output type: text or json (default text)
        +  -p, --profile string   ~/.databrickscfg profile
        +  -t, --target string    bundle target to use (if applicable)
        +
        +Use "databricks pipelines [command] --help" for more information about a command.
        +

latest (v1.25.0): same output as main (688c228)

TestAccept/pipelines/test/results/passed: branch ✅ | main (688c228) ❌ | latest (v1.25.0) ❌

main (688c228):

acceptance_test.go:1345: Diff:
        --- pipelines/test/results/passed/output.txt
        +++ /tmp/TestAcceptpipelinestestresultspassed3395706045/002/output.txt
        @@ -1,30 +1,55 @@
         
         === All tests pass
        ->>> [CLI] pipelines test --pipeline-id [UUID][0]
        -Pipeline: [UUID][0]
        -Update: [UUID][1]
        -Running tests...
        -1 of 2 PASSED tests/test_orders.py::test_valid_order ........................... [PASSED in 1.84s]
        -2 of 2 PASSED tests/test_orders.py::test_backfill .............................. [PASSED in 1.00s]
        +>>> [CLI] pipelines test --pipeline-id [UUID]
        +Error: unknown flag: --pipeline-id
         
        -Done. PASSED=2 FAILED=0 SKIPPED=0 ERROR=0 TOTAL=2
        +Usage:
        +  databricks pipelines [flags]
        +  databricks pipelines [command]
         
        ->>> print_requests.py --get //events
        -{
        -  "method": "GET",
        -  "path": "/api/2.0/pipelines/[UUID][0]/events",
        -  "q": {
        -    "filter": "update_id = '[UUID][1]'",
        -    "max_results": "250",
        -    "order_by": "timestamp asc"
        -  }
        -}
        -{
        -  "method": "GET",
        -  "path": "/api/2.0/pipelines/[UUID][0]/events",
        -  "q": {
        -    "filter": "update_id = '[UUID][1]'",
        -    "max_results": "250",
        -    "order_by": "timestamp asc"
        -  }
        -}
        +Main Commands
        +  deploy                Deploy pipelines
        +  destroy               Destroy a pipelines project
        +  dry-run               Validate correctness of the pipeline's graph
        +  generate              Generate pipeline configuration
        +  history               Retrieve past runs for a pipeline
        +  init                  Initialize a new pipelines project
        +  logs                  Retrieve events for a pipeline
        +  open                  Open a pipeline in the browser
        +  run                   Run a pipeline
        +  show                  Preview a table's columns and sample rows
        +  stop                  Stop a pipeline
        +
        +Management Commands
        +  apply-environment     *Public Preview* Apply the latest environment to the pipeline.
        +  clone                 Clone a pipeline.
        +  create                Create a pipeline.
        +  delete                Delete a pipeline.
        +  get                   Get a pipeline.
        +  get-update            Get a pipeline update.
        +  list-pipeline-events  List pipeline events.
        +  list-pipelines        List pipelines.
        +  list-updates          List pipeline updates.
        +  start-update          Start a pipeline.
        +  update                Edit a pipeline.
        +
        +Permission Commands
        +  get-permission-levels Get pipeline permission levels.
        +  get-permissions       Get pipeline permissions.
        +  set-permissions       Set pipeline permissions.
        +  update-permissions    Update pipeline permissions.
        +
        +Flags:
        +  -h, --help          help for pipelines
        +      --var strings   set values for variables defined in project config. Example: --var="foo=bar"
        +
        +Global Flags:
        +      --debug            enable debug logging
        +  -o, --output type      output type: text or json (default text)
        +  -p, --profile string   ~/.databrickscfg profile
        +  -t, --target string    bundle target to use (if applicable)
        +
        +Use "databricks pipelines [command] --help" for more information about a command.
        +
        +
        +Exit code: 1

latest (v1.25.0): same output as main (688c228)

TestAccept/pipelines/test/selection: branch ✅ | main (688c228) ❌ | latest (v1.25.0) ❌

main (688c228):

acceptance_test.go:1345: Diff:
        --- pipelines/test/selection/output.txt
        +++ /tmp/TestAcceptpipelinestestselection2440255433/002/output.txt
        @@ -4,80 +4,59 @@
         Created pipelines.my_pipeline
         Files: 3 uploaded, 0 deleted
         Resources: 1 created, 0 changed, 0 deleted, 0 unchanged
        -View your pipeline my_pipeline here: [DATABRICKS_URL]/pipelines/[UUID][0]?w=[WORKSPACE_ID]
        +View your pipeline my_pipeline here: [DATABRICKS_URL]/pipelines/[UUID]?w=[WORKSPACE_ID]
         
         === Run all tests of the only pipeline
         >>> [CLI] pipelines test --no-wait
        -Recommendation: This command runs the last deployed version of the code
        +Error: unknown flag: --no-wait
         
        -If you've made local changes, run 'databricks pipelines deploy' first to ensure they are included.
        +Usage:
        +  databricks pipelines [flags]
        +  databricks pipelines [command]
         
        -Pipeline: my_pipeline ([UUID][0])
        -Update: [UUID][1]
        +Main Commands
        +  deploy                Deploy pipelines
        +  destroy               Destroy a pipelines project
        +  dry-run               Validate correctness of the pipeline's graph
        +  generate              Generate pipeline configuration
        +  history               Retrieve past runs for a pipeline
        +  init                  Initialize a new pipelines project
        +  logs                  Retrieve events for a pipeline
        +  open                  Open a pipeline in the browser
        +  run                   Run a pipeline
        +  show                  Preview a table's columns and sample rows
        +  stop                  Stop a pipeline
         
        ->>> print_requests.py //updates
        -{
        -  "method": "POST",
        -  "path": "/api/2.0/pipelines/[UUID][0]/updates",
        -  "body": {
        -    "test_only": true
        -  }
        -}
        +Management Commands
        +  apply-environment     *Public Preview* Apply the latest environment to the pipeline.
        +  clone                 Clone a pipeline.
        +  create                Create a pipeline.
        +  delete                Delete a pipeline.
        +  get                   Get a pipeline.
        +  get-update            Get a pipeline update.
        +  list-pipeline-events  List pipeline events.
        +  list-pipelines        List pipelines.
        +  list-updates          List pipeline updates.
        +  start-update          Start a pipeline.
        +  update                Edit a pipeline.
         
        -=== Select tests by relative path and test ID, with pytest arguments in both --pytest-arg forms
        ->>> MSYS_NO_PATHCONV=1 [CLI] pipelines test my_pipeline tests tests/test_orders.py::test_backfill /Workspace/Users/someone@example.com/tests/test_other.py --pytest-arg=-k --pytest-arg=orders and not slow --pytest-arg -m --pytest-arg smoke,fast --no-wait
        -Recommendation: This command runs the last deployed version of the code
        +Permission Commands
        +  get-permission-levels Get pipeline permission levels.
        +  get-permissions       Get pipeline permissions.
        +  set-permissions       Set pipeline permissions.
        +  update-permissions    Update pipeline permissions.
         
        -If you've made local changes, run 'databricks pipelines deploy' first to ensure they are included.
        +Flags:
        +  -h, --help          help for pipelines
        +      --var strings   set values for variables defined in project config. Example: --var="foo=bar"
         
        -Pipeline: my_pipeline ([UUID][0])
        -Update: [UUID][2]
        +Global Flags:
        +      --debug            enable debug logging
        +  -o, --output type      output type: text or json (default text)
        +  -p, --profile string   ~/.databrickscfg profile
        +  -t, --target string    bundle target to use (if applicable)
         
        ->>> print_requests.py //updates
        -{
        -  "method": "POST",
        -  "path": "/api/2.0/pipelines/[UUID][0]/updates",
        -  "body": {
        -    "test_only": true,
        -    "test_options": {
        -      "test_selection": [
        -        "/Workspace/Users/[USERNAME]/.bundle/test-pipeline-test-selection/default/files/tests",
        -        "/Workspace/Users/[USERNAME]/.bundle/test-pipeline-test-selection/default/files/tests/test_orders.py::test_backfill",
        -        "/Workspace/Users/someone@example.com/tests/test_other.py"
        -      ],
        -      "pytest_args": [
        -        "-k",
        -        "orders and not slow",
        -        "-m",
        -        "smoke,fast"
        -      ]
        -    }
        -  }
        -}
        +Use "databricks pipelines [command] --help" for more information about a command.
         
        -=== Relative paths resolve from the working directory
        ->>> [CLI] pipelines test my_pipeline test_orders.py::test_valid_order --no-wait
        -Recommendation: This command runs the last deployed version of the code
         
        -If you've made local changes, run 'databricks pipelines deploy' first to ensure they are included.
        -
        -Pipeline: my_pipeline ([UUID][0])
        -Update: [UUID][3]
        -
        ->>> print_requests.py //updates
        -{
        -  "method": "POST",
        -  "path": "/api/2.0/pipelines/[UUID][0]/updates",
        -  "body": {
        -    "test_only": true,
        -    "test_options": {
        -      "test_selection": [
        -        "/Workspace/Users/[USERNAME]/.bundle/test-pipeline-test-selection/default/files/tests/test_orders.py::test_valid_order"
        -      ]
        -    }
        -  }
        -}
        -
        -=== Test selections require the pipeline KEY first
        ->>> musterr [CLI] pipelines test tests/test_orders.py
        -Error: "tests/test_orders.py" is not a pipeline in this bundle (pipelines: my_pipeline). Usage: databricks pipelines test my_pipeline tests/test_orders.py
        +Exit code: 1

latest (v1.25.0): same output as main (688c228)

TestAccept/pipelines/test/submit-error: branch ✅ | main (688c228) ❌ | latest (v1.25.0) ❌

main (688c228):

acceptance_test.go:1345: Diff:
        --- pipelines/test/submit-error/output.txt
        +++ /tmp/TestAcceptpipelinestestsubmit-error3589498520/002/output.txt
        @@ -1,13 +1,58 @@
         
         === A submit error from the service is shown as is, and no update ID is printed
        ->>> musterr [CLI] pipelines test --pipeline-id [UUID][0]
        -Error: An active update '[UUID][1]' already exists for pipeline '[UUID][2]'.
        +>>> musterr [CLI] pipelines test --pipeline-id [UUID]
        +Error: unknown flag: --pipeline-id
         
        +Usage:
        +  databricks pipelines [flags]
        +  databricks pipelines [command]
        +
        +Main Commands
        +  deploy                Deploy pipelines
        +  destroy               Destroy a pipelines project
        +  dry-run               Validate correctness of the pipeline's graph
        +  generate              Generate pipeline configuration
        +  history               Retrieve past runs for a pipeline
        +  init                  Initialize a new pipelines project
        +  logs                  Retrieve events for a pipeline
        +  open                  Open a pipeline in the browser
        +  run                   Run a pipeline
        +  show                  Preview a table's columns and sample rows
        +  stop                  Stop a pipeline
        +
        +Management Commands
        +  apply-environment     *Public Preview* Apply the latest environment to the pipeline.
        +  clone                 Clone a pipeline.
        +  create                Create a pipeline.
        +  delete                Delete a pipeline.
        +  get                   Get a pipeline.
        +  get-update            Get a pipeline update.
        +  list-pipeline-events  List pipeline events.
        +  list-pipelines        List pipelines.
        +  list-updates          List pipeline updates.
        +  start-update          Start a pipeline.
        +  update                Edit a pipeline.
        +
        +Permission Commands
        +  get-permission-levels Get pipeline permission levels.
        +  get-permissions       Get pipeline permissions.
        +  set-permissions       Set pipeline permissions.
        +  update-permissions    Update pipeline permissions.
        +
        +Flags:
        +  -h, --help          help for pipelines
        +      --var strings   set values for variables defined in project config. Example: --var="foo=bar"
        +
        +Global Flags:
        +      --debug            enable debug logging
        +  -o, --output type      output type: text or json (default text)
        +  -p, --profile string   ~/.databrickscfg profile
        +  -t, --target string    bundle target to use (if applicable)
        +
        +Use "databricks pipelines [command] --help" for more information about a command.
        +
        +
         >>> print_requests.py //updates
        -{
        -  "method": "POST",
        -  "path": "/api/2.0/pipelines/[UUID][0]/updates",
        -  "body": {
        -    "test_only": true
        -  }
        -}
        +File [OUTPUT_DIR]/out.requests.txt not found
        +
        +Exit code: 1

latest (v1.25.0): same output as main (688c228)

@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Nov 2, 2026 •

Copy link
Copy Markdown
Collaborator

✅ Integration test report

Commit: b45969f

Run: 42728311165

Env ✅​pass 🙈​skip Time
✅​ aws linux-2core-8gb 271 23 5:05
✅​ aws-windows-latest-4core-16gb 273 21 4:58
✅​ azure linux-2core-8gb 270 23 4:45
✅​ azure-windows-latest-4core-16gb 272 21 5:48
✅​ gcp linux-2core-8gb 271 23 5:22
✅​ gcp-windows-latest-4core-16gb 273 21 6:48
Top 3 slowest tests (at least 2 minutes):
duration env testname
3:14 gcp-windows-latest-4core-16gb TestAccept
3:06 aws-windows-latest-4core-16gb TestAccept
3:03 azure-windows-latest-4core-16gb TestAccept

@tharunbhuk-db
tharunbhuk-db force-pushed the tharun/pipelines-test-pr2-output branch from 8542365 to cb0a526 Compare November 2, 2026 23:22
@tharunbhuk-db
tharunbhuk-db force-pushed the tharun/pipelines-test-pr2-output branch 2 times, most recently from 025649f to 23e0bb0 Compare November 3, 2026 17:34
Comment thread cmd/pipelines/test.go Outdated
if incompleteReason == "" {
summary := run.summaries[0]
testsFailed := summary.Total == 0 || summary.Failed > 0 || summary.Errors > 0
if !testsFailed && state == pipelines.UpdateInfoStateCompleted {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A COMPLETED update whose tests all passed returns nil here without checking run.fatalError, so an update that also reported a fatal error exits 0. Could we add run.fatalError == "" to this condition, with a TestTestRunError case for it?

Related: a run where every test is SKIPPED also returns nil here. That matches the design (an all-skipped run completes), but CI and agents will read it as green, so the help should say so explicitly.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. testRunError also checks run.fatalError == "", with a test case. The help says an all-skipped run exits 0.

Comment thread cmd/pipelines/test.go
UpdateId: tw.updateID,
})
if err != nil {
return "", err

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A GetUpdate error (and the same for the events read just below) ends the wait with the raw error and no --resume hint, even though the update keeps running. The SDK client already retries transient HTTP errors, so no extra retry logic is needed here. Could the error just include the same resume command as the timeout path?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll do this in a follow-up.

Comment thread cmd/pipelines/test.go Outdated
updateID := resumeUpdateID
if !resume {
req := newStartTestUpdateRequest(target.selections, pytestArgs)
updateID, err = startTestUpdate(ctx, target.w, target.pipelineID, req)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This POST uses the signal context, so a Ctrl+C during Start Update loses the update ID, and the update runs with no way to find it from this output. Could the POST use cmd.Context()?

Also on this line: when the service rejects the start because an update is already active, a short static hint after its message would give agents the next step, for example "Run 'databricks pipelines stop KEY' to cancel it, or wait for it to finish." There's no need to parse the ID out of the message.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in #7426: Start Update uses cmd.Context(), tested by TestInterruptDuringStartUpdateKeepsTheUpdateID. The already-active hint: I'll do that in a follow-up.

Comment thread cmd/pipelines/test_render.go Outdated
fmt.Fprintln(out)

if incompleteReason != "" {
fmt.Fprintf(out, "RUN INCOMPLETE: %s.\n", incompleteReason)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The most likely cause of INCOMPLETE is events still arriving after the terminal state, and the update can still be read later. Could we print the --resume command here too?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll do this in a follow-up.


// testVerdictErrorPrefix starts the message of the error by which the service reports that tests
// failed, such as "[PIPELINE_TEST_FAILED] 1 of the pipeline's tests failed (1 passed, 0 skipped).".
const testVerdictErrorPrefix = "[PIPELINE_TEST_FAILED]"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Recognizing the service verdict by message prefix is fragile. The exit code doesn't depend on it, only the output does. The SDK's ErrorDetail has no error class yet, so a comment saying this is temporary until a class is exposed would help.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in #7432. A comment says the message is matched because ErrorDetail has no error class yet.

Comment thread cmd/pipelines/test.go
// update ended in another way or reports an error of its own. The service's own verdict,
// such as "[PIPELINE_TEST_FAILED] 1 of the pipeline's tests failed", is not.
ended := state == pipelines.UpdateInfoStateFailed || state == pipelines.UpdateInfoStateCompleted
if testsFailed && ended && run.fatalError == "" {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Every failure path exits 1. Test failures and zero-collected runs print nothing to stderr, while infra failures print Error:, so an agent tells "fix the code" from "retry" only from the output. pipelines run behaves the same, so distinct exit codes would be a CLI-wide decision. For this stack, could the help just say what exit 0 and 1 mean? Small related nit: with --junit-xml, the JUnit report: line prints after the summary, so tail -1 misses the summary; printing it before the summary would keep the summary last.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. The help says what exit 0 and 1 mean. The JUnit report: line prints before the summary from #7434.

summary := run.summaries[0]
switch summary.Total {
case 0:
fmt.Fprintln(out, "No tests were collected.")

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agents hit this often, and the message gives no hint. Naming what was run would make it self-correcting, e.g. No tests were collected from <testfile>. Check the test selection and any --pytest-arg filters. (I'd avoid restating pytest's naming rules here, since projects can change them in their pytest configuration.) But if we can give a bit better message that would be nice.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll do this in a follow-up.

testCasePassed testCaseStatus = "PASSED"
testCaseFailed testCaseStatus = "FAILED"
testCaseSkipped testCaseStatus = "SKIPPED"
testCaseError testCaseStatus = "ERROR"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should this be errored? I see we do error and errored at some places would be good if we can do consistent.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These follow the proto: the case status is ERROR and the summary field is errored. The summary line uses the status names (ERROR=).

@@ -0,0 +1,799 @@
# 200 cases emit 601 events, more than two pages of List Pipeline Events.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: instead of committing 800 lines of identical tests, could the script generate this file before pipelines deploy? The fake collects tests with a regex over the deployed files (libs/testserver/pipelines_testing.go:106), so this works the same and runs on bash 3.2 and Git Bash:

for i in $(seq 0 199); do printf 'def test_case_%03d():\n    assert True\n\n' "$i"; done > tests/test_many.py

Also worth checking how many cases you actually need. The CLI doesn't send max_results, so the fake pages at its default of 25 events (pipelines_testing.go:332), and the output shows 8 pages read. Fewer cases may cover the same paths. The page math reads better in the script than as a comment in a generated file.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. The unit test TestReadTestEvents200Cases in #7432 replaces it. The reader sends max_results=250.

assert.Equal(t, 400, response.StatusCode)
}

func TestPipelineStartUpdateRejectsInvalidTestOptions(t *testing.T) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: this file tests the fake more deeply than the CLI contract needs, and a few could go:

  • TestPipelineStartUpdateRejectsInvalidTestOptions and TestPipelineStartUpdateRejectsTestOptionsWithoutTestOnly, together with the fake's pipelineTestOptionsError. The CLI validates --pytest-arg and makes selections absolute before any request, and it always sends test_only: true, so pipelines test never reaches this validation.
  • TestPipelineFiniteUpdateIsActiveUntilItEnds, TestPipelineUpdateThatIsNeverPolledEndsWithItsFirstState and TestPipelineStopHidesEventsOfStatesNotReached. They pin the fake's state timing, and what the CLI sees (409, CANCELED) is golden in run-control and submit-error.
  • TestPipelineEventsPages. Paging is covered by the CLI's own reader test and many-cases.

TestCollectPipelineTests, TestPipelineTestUpdate and TestPipelineTestUpdateConflictAndStop cover what the acceptance tests rely on. Also, line 242 repeats the assertion on line 241.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. The stack no longer changes libs/testserver.

@tharunbhuk-db
tharunbhuk-db marked this pull request as ready for review November 5, 2026 16:09
@tharunbhuk-db tharunbhuk-db changed the title [WIP] Wait for and print the results of databricks pipelines test Wait for and print the results of databricks pipelines test Nov 5, 2026
@tharunbhuk-db
tharunbhuk-db marked this pull request as draft November 9, 2026 23:57
@tharunbhuk-db
tharunbhuk-db force-pushed the tharun/pipelines-test-pr1-command branch from 66e7f7e to 62049a1 Compare November 10, 2026 00:01
@tharunbhuk-db
tharunbhuk-db changed the base branch from tharun/pipelines-test-pr1-command to tharun/pipelines-test-a2-fake-server November 10, 2026 00:25
@tharunbhuk-db
tharunbhuk-db force-pushed the tharun/pipelines-test-pr2-output branch from 23e0bb0 to b45969f Compare November 10, 2026 00:25
tharunbhuk-db added a commit that referenced this pull request Nov 10, 2026
This commit adds the changes of #7297, #7425, #7298, #7426 and #7299, so that
the next commit can build on both stacks. It is not for review here, and goes
away when that stack merges.
@tharunbhuk-db
tharunbhuk-db force-pushed the tharun/pipelines-test-pr2-output branch from b45969f to e8a772a Compare November 10, 2026 04:14
@tharunbhuk-db
tharunbhuk-db changed the base branch from tharun/pipelines-test-a2-fake-server to tharun/pipelines-test-print-results November 10, 2026 04:14
@tharunbhuk-db
tharunbhuk-db force-pushed the tharun/pipelines-test-print-results branch 2 times, most recently from 8329c9f to 2036088 Compare November 10, 2026 06:25
@tharunbhuk-db
tharunbhuk-db force-pushed the tharun/pipelines-test-pr2-output branch from e8a772a to e7de143 Compare November 10, 2026 06:26
@tharunbhuk-db

Copy link
Copy Markdown
Contributor Author

Follow-ups from review:

  • Add the --resume command to errors from Get Update and the events read while waiting. (thread)
  • When Start Update fails because an update is already active, add a hint: "Run 'databricks pipelines stop KEY' to cancel it, or wait for it to finish." (thread)
  • Print the --resume command for an incomplete run. (thread)
  • Name the selection in "No tests were collected", and point to the test selection and --pytest-arg filters. (thread)

@tharunbhuk-db
tharunbhuk-db marked this pull request as ready for review November 10, 2026 06:54
@tharunbhuk-db
tharunbhuk-db requested a review from a team as a code owner November 10, 2026 06:54
@tharunbhuk-db
tharunbhuk-db requested review from pranshupand-db and removed request for a team November 10, 2026 06:54
The command waits for the test-only update, prints each test case as it
finishes, then the failure details and a summary line. It exits with code 0
only for a complete run where at least one test ran and none failed or errored,
and a completed update that reports a fatal error does not pass. Test IDs are
shown relative to the project folder. With the no-wait flag, the command returns
after the update starts.

The wait takes the steps of the bundle pipeline runner's wait loop. The
acceptance tests stub the service per case, as the other pipelines tests do.

Co-authored-by: Isaac <no-reply@databricks.com>
@tharunbhuk-db
tharunbhuk-db force-pushed the tharun/pipelines-test-print-results branch from 2036088 to 42a9409 Compare November 10, 2026 17:13
@tharunbhuk-db
tharunbhuk-db force-pushed the tharun/pipelines-test-pr2-output branch from e7de143 to d0d7138 Compare November 10, 2026 17:13

@pranshupand-db pranshupand-db left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Follow-ups for the stack are in the #7297 review: #7297

idelafuente-oc pushed a commit to idelafuente-oc/cli that referenced this pull request Nov 11, 2026
## 🥞 Stacked PR
Use this [link](https://github.com/databricks/cli/pull/7297/files) to
review incremental changes.
-
[**stack/pipelines-test-pr1-command**](databricks#7297)
[[Files changed](https://github.com/databricks/cli/pull/7297/files)] ←
_this PR_
-
[stack/pipelines-test-bundle-key](databricks#7431)
[[Files changed](https://github.com/databricks/cli/pull/7431/files)]
-
[stack/pipelines-test-read-events](databricks#7432)
[[Files changed](https://github.com/databricks/cli/pull/7432/files)]
-
[stack/pipelines-test-print-results](databricks#7433)
[[Files changed](https://github.com/databricks/cli/pull/7433/files)]
-
[stack/pipelines-test-pr2-output](databricks#7298)
[[Files changed](https://github.com/databricks/cli/pull/7298/files)]
-
[stack/pipelines-test-a4-run-control](databricks#7426)
[[Files changed](https://github.com/databricks/cli/pull/7426/files)]
-
[stack/pipelines-test-pr3-junit](databricks#7299)
[[Files changed](https://github.com/databricks/cli/pull/7299/files)]
-
[stack/pipelines-test-junit-flag](databricks#7434)
[[Files changed](https://github.com/databricks/cli/pull/7434/files)]

---------
- Add `databricks pipelines test --pipeline-id <id>
[TEST_SELECTION...]`: starts a test-only update and prints its IDs.
- Experimental: runs only with `DATABRICKS_CLI_PIPELINES_TEST=true`.
- Selections are workspace paths or paths relative to `root_path`, sent
as typed. `--pytest-arg` accepts only `-k` and `-m`.

Tests: unit tests; acceptance in
`acceptance/pipelines/test/{arguments,pipeline-id,submit-error}`.

This pull request and its description were written by Isaac.

Co-authored-by: Isaac <no-reply@databricks.com>
idelafuente-oc pushed a commit to idelafuente-oc/cli that referenced this pull request Nov 11, 2026
## 🥞 Stacked PR
-
[stack/pipelines-test-pr1-command](databricks#7297)
[[Files changed](https://github.com/databricks/cli/pull/7297/files)]
-
[**stack/pipelines-test-bundle-key**](databricks#7431)
[[Files changed](https://github.com/databricks/cli/pull/7431/files)] ←
_this PR_
-
[stack/pipelines-test-read-events](databricks#7432)
[[Files changed](https://github.com/databricks/cli/pull/7432/files)]
-
[stack/pipelines-test-print-results](databricks#7433)
[[Files changed](https://github.com/databricks/cli/pull/7433/files)]
-
[stack/pipelines-test-pr2-output](databricks#7298)
[[Files changed](https://github.com/databricks/cli/pull/7298/files)]
-
[stack/pipelines-test-a4-run-control](databricks#7426)
[[Files changed](https://github.com/databricks/cli/pull/7426/files)]
-
[stack/pipelines-test-pr3-junit](databricks#7299)
[[Files changed](https://github.com/databricks/cli/pull/7299/files)]
-
[stack/pipelines-test-junit-flag](databricks#7434)
[[Files changed](https://github.com/databricks/cli/pull/7434/files)]

---------
- In a bundle, identify the pipeline by KEY: `pipelines test [flags]
[KEY] [TEST_SELECTION...]`.
- Relative selections are local paths, mapped to their deployed paths.
- A first argument that is not a KEY gets an error that names the
bundle's pipelines and the working command.

Tests: unit tests; acceptance in
`acceptance/pipelines/test/{selection,immutable}`.

This pull request and its description were written by Isaac.

Co-authored-by: Isaac <no-reply@databricks.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants