Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions acceptance/pipelines/test/arguments/output.txt
Original file line number Diff line number Diff line change
Expand Up @@ -23,10 +23,16 @@ example:

databricks pipelines test my_pipeline --pytest-arg=-k --pytest-arg='orders and not slow'

The command waits for the results. It exits with code 0 only if at least one
test was collected and none failed or errored, so a run whose tests were all
skipped succeeds; otherwise it exits with code 1. --no-wait returns after the
update starts. To stop the update, run 'pipelines stop'.

Usage:
databricks pipelines test [flags] [KEY] [TEST_SELECTION...]

Pipeline Test Flags:
--no-wait Don't wait for the tests to complete.
--pipeline-id string ID of the pipeline to test, instead of a KEY.
--pytest-arg stringArray Argument to pass to pytest (repeatable). Only -k and -m are supported.

Expand Down
4 changes: 2 additions & 2 deletions acceptance/pipelines/test/immutable/output.txt
Original file line number Diff line number Diff line change
Expand Up @@ -15,15 +15,15 @@ If you've made local changes, run 'databricks pipelines deploy' first to ensure
Error: cannot resolve test selection "tests/test_orders.py::test_backfill": local test paths are not supported with workspace.sync_mode immutable; pass an absolute workspace path

=== Absolute workspace paths and running all tests still work
>>> MSYS_NO_PATHCONV=1 [CLI] pipelines test my_pipeline /Workspace/Users/someone@example.com/tests/test_orders.py::test_backfill
>>> MSYS_NO_PATHCONV=1 [CLI] pipelines test my_pipeline /Workspace/Users/someone@example.com/tests/test_orders.py::test_backfill --no-wait
Recommendation: This command runs the last deployed version of the code

If you've made local changes, run 'databricks pipelines deploy' first to ensure they are included.

Pipeline: my_pipeline ([UUID][0])
Update: [UUID][1]

>>> [CLI] pipelines test my_pipeline
>>> [CLI] pipelines test my_pipeline --no-wait
Recommendation: This command runs the last deployed version of the code

If you've made local changes, run 'databricks pipelines deploy' first to ensure they are included.
Expand Down
4 changes: 2 additions & 2 deletions acceptance/pipelines/test/immutable/script
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,6 @@ title "Local test paths are not supported with workspace.sync_mode immutable"
trace musterr $CLI pipelines test my_pipeline tests/test_orders.py::test_backfill

title "Absolute workspace paths and running all tests still work"
trace MSYS_NO_PATHCONV=1 $CLI pipelines test my_pipeline /Workspace/Users/someone@example.com/tests/test_orders.py::test_backfill
trace $CLI pipelines test my_pipeline
trace MSYS_NO_PATHCONV=1 $CLI pipelines test my_pipeline /Workspace/Users/someone@example.com/tests/test_orders.py::test_backfill --no-wait
trace $CLI pipelines test my_pipeline --no-wait
trace print_requests.py //updates
4 changes: 2 additions & 2 deletions acceptance/pipelines/test/pipeline-id/output.txt
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@

=== Relative test selections resolve against the pipeline root_path
>>> MSYS_NO_PATHCONV=1 [CLI] pipelines test --pipeline-id [UUID][0] tests/test_orders.py::test_backfill /Workspace/Users/someone@example.com/other/tests
>>> MSYS_NO_PATHCONV=1 [CLI] pipelines test --pipeline-id [UUID][0] tests/test_orders.py::test_backfill /Workspace/Users/someone@example.com/other/tests --no-wait
Pipeline: [UUID][0]
Update: [UUID][1]

Expand All @@ -23,7 +23,7 @@ Update: [UUID][1]
>>> musterr [CLI] pipelines test --pipeline-id [UUID][2] tests
Error: cannot resolve test selection "tests": the pipeline has no root_path; use an absolute workspace path such as /Workspace/Users/someone@example.com/tests

>>> MSYS_NO_PATHCONV=1 [CLI] pipelines test --pipeline-id [UUID][2] /Workspace/Users/someone@example.com/tests/test_other.py::test_x
>>> MSYS_NO_PATHCONV=1 [CLI] pipelines test --pipeline-id [UUID][2] /Workspace/Users/someone@example.com/tests/test_other.py::test_x --no-wait
Pipeline: [UUID][2]
Update: [UUID][3]

Expand Down
4 changes: 2 additions & 2 deletions acceptance/pipelines/test/pipeline-id/script
Original file line number Diff line number Diff line change
Expand Up @@ -3,12 +3,12 @@ no_root_pipeline_id=$($CLI pipelines create --json '{"name": "no-root"}' | jq -r
rm "$OUT_REQUESTS"

title "Relative test selections resolve against the pipeline root_path"
trace MSYS_NO_PATHCONV=1 $CLI pipelines test --pipeline-id $pipeline_id tests/test_orders.py::test_backfill /Workspace/Users/someone@example.com/other/tests
trace MSYS_NO_PATHCONV=1 $CLI pipelines test --pipeline-id $pipeline_id tests/test_orders.py::test_backfill /Workspace/Users/someone@example.com/other/tests --no-wait
trace print_requests.py //updates

title "Without a root_path, test selections must be workspace paths"
trace musterr $CLI pipelines test --pipeline-id $no_root_pipeline_id tests
trace MSYS_NO_PATHCONV=1 $CLI pipelines test --pipeline-id $no_root_pipeline_id /Workspace/Users/someone@example.com/tests/test_other.py::test_x
trace MSYS_NO_PATHCONV=1 $CLI pipelines test --pipeline-id $no_root_pipeline_id /Workspace/Users/someone@example.com/tests/test_other.py::test_x --no-wait
trace print_requests.py //updates

title "Outside a project, --pipeline-id is required"
Expand Down
1 change: 1 addition & 0 deletions acceptance/pipelines/test/results/failed/out.test.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Cloud = false
20 changes: 20 additions & 0 deletions acceptance/pipelines/test/results/failed/output.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@

=== Failed, errored and skipped tests
>>> musterr [CLI] pipelines test --pipeline-id [UUID][0]
Pipeline: [UUID][0]
Update: [UUID][1]
Running tests...
1 of 4 PASSED tests/test_orders.py::test_valid_order ........................... [PASSED]
2 of 4 FAILED tests/test_orders.py::test_backfill .............................. [FAILED in 2.14s]
3 of 4 SKIPPED tests/test_orders.py::test_legacy_format ........................ [SKIPPED]
4 of 4 ERROR tests/test_orders.py::test_fixture ................................ [ERROR]

Completed with 1 failed and 1 errored tests:

Failure in tests/test_orders.py::test_backfill
AssertionError: assert 100 == 150

Error in tests/test_orders.py::test_fixture
RuntimeError: fixture orders setup failed

Done. PASSED=1 FAILED=1 SKIPPED=1 ERROR=1 TOTAL=4
2 changes: 2 additions & 0 deletions acceptance/pipelines/test/results/failed/script
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
title "Failed, errored and skipped tests"
trace musterr $CLI pipelines test --pipeline-id $pipeline_id
25 changes: 25 additions & 0 deletions acceptance/pipelines/test/results/failed/test.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
[[Server]]
Pattern = "POST /api/2.0/pipelines/{pipeline_id}/updates"
Response.Body = '{"update_id": "00000000-0000-0000-0000-0000000000a1"}'

[[Server]]
Pattern = "GET /api/2.0/pipelines/{pipeline_id}/updates/{update_id}"
Response.Body = '{"update": {"update_id": "00000000-0000-0000-0000-0000000000a1", "state": "FAILED"}}'

[[Server]]
Pattern = "GET /api/2.0/pipelines/{pipeline_id}/events"
Response.Body = '''
{"events": [
{"id": "e1", "event_type": "test_case_progress", "timestamp": "2026-01-01T00:00:00.000Z", "origin": {"test_case_id": "/Workspace/Users/someone@example.com/orders/tests/test_orders.py::test_valid_order"}, "details": {"test_case_progress": {"status": "PENDING"}}},
{"id": "e2", "event_type": "test_case_progress", "timestamp": "2026-01-01T00:00:00.000Z", "origin": {"test_case_id": "/Workspace/Users/someone@example.com/orders/tests/test_orders.py::test_backfill"}, "details": {"test_case_progress": {"status": "PENDING"}}},
{"id": "e3", "event_type": "test_case_progress", "timestamp": "2026-01-01T00:00:00.000Z", "origin": {"test_case_id": "/Workspace/Users/someone@example.com/orders/tests/test_orders.py::test_legacy_format"}, "details": {"test_case_progress": {"status": "PENDING"}}},
{"id": "e4", "event_type": "test_case_progress", "timestamp": "2026-01-01T00:00:00.000Z", "origin": {"test_case_id": "/Workspace/Users/someone@example.com/orders/tests/test_orders.py::test_fixture"}, "details": {"test_case_progress": {"status": "PENDING"}}},
{"id": "e5", "event_type": "test_case_progress", "timestamp": "2026-01-01T00:00:01.000Z", "origin": {"test_case_id": "/Workspace/Users/someone@example.com/orders/tests/test_orders.py::test_valid_order"}, "details": {"test_case_progress": {"status": "PASSED"}}},
{"id": "e6", "event_type": "test_case_progress", "timestamp": "2026-01-01T00:00:01.000Z", "origin": {"test_case_id": "/Workspace/Users/someone@example.com/orders/tests/test_orders.py::test_backfill"}, "details": {"test_case_progress": {"status": "RUNNING"}}},
{"id": "e7", "event_type": "test_case_progress", "timestamp": "2026-01-01T00:00:03.140Z", "origin": {"test_case_id": "/Workspace/Users/someone@example.com/orders/tests/test_orders.py::test_backfill"}, "details": {"test_case_progress": {"status": "FAILED"}}, "level": "ERROR", "error": {"fatal": false, "exceptions": [{"class_name": "AssertionError", "message": "assert 100 == 150"}]}},
{"id": "e8", "event_type": "test_case_progress", "timestamp": "2026-01-01T00:00:04.000Z", "origin": {"test_case_id": "/Workspace/Users/someone@example.com/orders/tests/test_orders.py::test_legacy_format"}, "details": {"test_case_progress": {"status": "SKIPPED"}}},
{"id": "e9", "event_type": "test_case_progress", "timestamp": "2026-01-01T00:00:05.000Z", "origin": {"test_case_id": "/Workspace/Users/someone@example.com/orders/tests/test_orders.py::test_fixture"}, "details": {"test_case_progress": {"status": "ERROR"}}, "level": "ERROR", "error": {"fatal": false, "exceptions": [{"class_name": "RuntimeError", "message": "fixture orders setup failed"}]}},
{"id": "e10", "event_type": "test_run_summary", "timestamp": "2026-01-01T00:00:06.000Z", "details": {"test_run_summary": {"total": 4, "passed": 1, "failed": 1, "skipped": 1, "errored": 1}}},
{"id": "e11", "event_type": "update_progress", "timestamp": "2026-01-01T00:00:07.000Z", "level": "ERROR", "error": {"fatal": true, "exceptions": [{"class_name": "TestFailure", "message": "[PIPELINE_TEST_FAILED] 2 of the pipeline's tests failed (1 passed, 1 skipped)."}]}}
]}
'''
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Cloud = false
11 changes: 11 additions & 0 deletions acceptance/pipelines/test/results/incomplete/output.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@

=== Compute ends before all results are received
>>> musterr [CLI] pipelines test --pipeline-id [UUID][0]
Pipeline: [UUID][0]
Update: [UUID][1]
Running tests...
1 of 2 PASSED tests/test_compute.py::test_small_order .......................... [PASSED]

RUN INCOMPLETE: the update failed before all test results were received.
Done. PASSED=1 FAILED=0 SKIPPED=0 ERROR=0 TOTAL=2
Error: update [UUID][1] is FAILED: ComputeTerminatedException: Compute terminated unexpectedly.
2 changes: 2 additions & 0 deletions acceptance/pipelines/test/results/incomplete/script
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
title "Compute ends before all results are received"
trace musterr $CLI pipelines test --pipeline-id $pipeline_id
19 changes: 19 additions & 0 deletions acceptance/pipelines/test/results/incomplete/test.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
[[Server]]
Pattern = "POST /api/2.0/pipelines/{pipeline_id}/updates"
Response.Body = '{"update_id": "00000000-0000-0000-0000-0000000000a1"}'

[[Server]]
Pattern = "GET /api/2.0/pipelines/{pipeline_id}/updates/{update_id}"
Response.Body = '{"update": {"update_id": "00000000-0000-0000-0000-0000000000a1", "state": "FAILED"}}'

[[Server]]
Pattern = "GET /api/2.0/pipelines/{pipeline_id}/events"
Response.Body = '''
{"events": [
{"id": "e1", "event_type": "test_case_progress", "timestamp": "2026-01-01T00:00:00.000Z", "origin": {"test_case_id": "/Workspace/Users/someone@example.com/orders/tests/test_compute.py::test_small_order"}, "details": {"test_case_progress": {"status": "PENDING"}}},
{"id": "e2", "event_type": "test_case_progress", "timestamp": "2026-01-01T00:00:00.000Z", "origin": {"test_case_id": "/Workspace/Users/someone@example.com/orders/tests/test_compute.py::test_large_order"}, "details": {"test_case_progress": {"status": "PENDING"}}},
{"id": "e3", "event_type": "test_case_progress", "timestamp": "2026-01-01T00:00:01.000Z", "origin": {"test_case_id": "/Workspace/Users/someone@example.com/orders/tests/test_compute.py::test_small_order"}, "details": {"test_case_progress": {"status": "PASSED"}}},
{"id": "e4", "event_type": "test_case_progress", "timestamp": "2026-01-01T00:00:01.000Z", "origin": {"test_case_id": "/Workspace/Users/someone@example.com/orders/tests/test_compute.py::test_large_order"}, "details": {"test_case_progress": {"status": "RUNNING"}}},
{"id": "e5", "event_type": "update_progress", "timestamp": "2026-01-01T00:00:02.000Z", "level": "ERROR", "error": {"fatal": true, "exceptions": [{"class_name": "ComputeTerminatedException", "message": "Compute terminated unexpectedly."}]}}
]}
'''
1 change: 1 addition & 0 deletions acceptance/pipelines/test/results/passed/out.test.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Cloud = false
30 changes: 30 additions & 0 deletions acceptance/pipelines/test/results/passed/output.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@

=== All tests pass
>>> [CLI] pipelines test --pipeline-id [UUID][0]
Pipeline: [UUID][0]
Update: [UUID][1]
Running tests...
1 of 2 PASSED tests/test_orders.py::test_valid_order ........................... [PASSED in 1.84s]
2 of 2 PASSED tests/test_orders.py::test_backfill .............................. [PASSED in 1.00s]

Done. PASSED=2 FAILED=0 SKIPPED=0 ERROR=0 TOTAL=2

>>> print_requests.py --get //events
{
"method": "GET",
"path": "/api/2.0/pipelines/[UUID][0]/events",
"q": {
"filter": "update_id = '[UUID][1]'",
"max_results": "250",
"order_by": "timestamp asc"
}
}
{
"method": "GET",
"path": "/api/2.0/pipelines/[UUID][0]/events",
"q": {
"filter": "update_id = '[UUID][1]'",
"max_results": "250",
"order_by": "timestamp asc"
}
}
3 changes: 3 additions & 0 deletions acceptance/pipelines/test/results/passed/script
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
title "All tests pass"
trace $CLI pipelines test --pipeline-id $pipeline_id
trace print_requests.py --get //events
21 changes: 21 additions & 0 deletions acceptance/pipelines/test/results/passed/test.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
[[Server]]
Pattern = "POST /api/2.0/pipelines/{pipeline_id}/updates"
Response.Body = '{"update_id": "00000000-0000-0000-0000-0000000000a1"}'

[[Server]]
Pattern = "GET /api/2.0/pipelines/{pipeline_id}/updates/{update_id}"
Response.Body = '{"update": {"update_id": "00000000-0000-0000-0000-0000000000a1", "state": "COMPLETED"}}'

[[Server]]
Pattern = "GET /api/2.0/pipelines/{pipeline_id}/events"
Response.Body = '''
{"events": [
{"id": "e1", "event_type": "test_case_progress", "timestamp": "2026-01-01T00:00:00.000Z", "origin": {"test_case_id": "/Workspace/Users/someone@example.com/orders/tests/test_orders.py::test_valid_order"}, "details": {"test_case_progress": {"status": "PENDING"}}},
{"id": "e2", "event_type": "test_case_progress", "timestamp": "2026-01-01T00:00:00.000Z", "origin": {"test_case_id": "/Workspace/Users/someone@example.com/orders/tests/test_orders.py::test_backfill"}, "details": {"test_case_progress": {"status": "PENDING"}}},
{"id": "e3", "event_type": "test_case_progress", "timestamp": "2026-01-01T00:00:01.000Z", "origin": {"test_case_id": "/Workspace/Users/someone@example.com/orders/tests/test_orders.py::test_valid_order"}, "details": {"test_case_progress": {"status": "RUNNING"}}},
{"id": "e4", "event_type": "test_case_progress", "timestamp": "2026-01-01T00:00:02.840Z", "origin": {"test_case_id": "/Workspace/Users/someone@example.com/orders/tests/test_orders.py::test_valid_order"}, "details": {"test_case_progress": {"status": "PASSED"}}},
{"id": "e5", "event_type": "test_case_progress", "timestamp": "2026-01-01T00:00:03.000Z", "origin": {"test_case_id": "/Workspace/Users/someone@example.com/orders/tests/test_orders.py::test_backfill"}, "details": {"test_case_progress": {"status": "RUNNING"}}},
{"id": "e6", "event_type": "test_case_progress", "timestamp": "2026-01-01T00:00:04.000Z", "origin": {"test_case_id": "/Workspace/Users/someone@example.com/orders/tests/test_orders.py::test_backfill"}, "details": {"test_case_progress": {"status": "PASSED"}}},
{"id": "e7", "event_type": "test_run_summary", "timestamp": "2026-01-01T00:00:05.000Z", "details": {"test_run_summary": {"total": 2, "passed": 2, "failed": 0, "skipped": 0, "errored": 0}}}
]}
'''
3 changes: 3 additions & 0 deletions acceptance/pipelines/test/results/script.prepare
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
# Each case stubs Start Update, Get Update and List Pipeline Events in its test.toml.
pipeline_id=$($CLI pipelines create --json '{"name": "orders", "root_path": "/Workspace/Users/someone@example.com/orders"}' | jq -r .pipeline_id)
rm "$OUT_REQUESTS"
6 changes: 3 additions & 3 deletions acceptance/pipelines/test/selection/output.txt
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ Resources: 1 created, 0 changed, 0 deleted, 0 unchanged
View your pipeline my_pipeline here: [DATABRICKS_URL]/pipelines/[UUID][0]?w=[WORKSPACE_ID]

=== Run all tests of the only pipeline
>>> [CLI] pipelines test
>>> [CLI] pipelines test --no-wait
Recommendation: This command runs the last deployed version of the code

If you've made local changes, run 'databricks pipelines deploy' first to ensure they are included.
Expand All @@ -25,7 +25,7 @@ Update: [UUID][1]
}

=== Select tests by relative path and test ID, with pytest arguments in both --pytest-arg forms
>>> MSYS_NO_PATHCONV=1 [CLI] pipelines test my_pipeline tests tests/test_orders.py::test_backfill /Workspace/Users/someone@example.com/tests/test_other.py --pytest-arg=-k --pytest-arg=orders and not slow --pytest-arg -m --pytest-arg smoke,fast
>>> MSYS_NO_PATHCONV=1 [CLI] pipelines test my_pipeline tests tests/test_orders.py::test_backfill /Workspace/Users/someone@example.com/tests/test_other.py --pytest-arg=-k --pytest-arg=orders and not slow --pytest-arg -m --pytest-arg smoke,fast --no-wait
Recommendation: This command runs the last deployed version of the code

If you've made local changes, run 'databricks pipelines deploy' first to ensure they are included.
Expand Down Expand Up @@ -56,7 +56,7 @@ Update: [UUID][2]
}

=== Relative paths resolve from the working directory
>>> [CLI] pipelines test my_pipeline test_orders.py::test_valid_order
>>> [CLI] pipelines test my_pipeline test_orders.py::test_valid_order --no-wait
Recommendation: This command runs the last deployed version of the code

If you've made local changes, run 'databricks pipelines deploy' first to ensure they are included.
Expand Down
6 changes: 3 additions & 3 deletions acceptance/pipelines/test/selection/script
Original file line number Diff line number Diff line change
Expand Up @@ -2,16 +2,16 @@ trace $CLI pipelines deploy
rm "$OUT_REQUESTS"

title "Run all tests of the only pipeline"
trace $CLI pipelines test
trace $CLI pipelines test --no-wait
trace print_requests.py //updates

title "Select tests by relative path and test ID, with pytest arguments in both --pytest-arg forms"
trace MSYS_NO_PATHCONV=1 $CLI pipelines test my_pipeline tests tests/test_orders.py::test_backfill /Workspace/Users/someone@example.com/tests/test_other.py --pytest-arg=-k --pytest-arg='orders and not slow' --pytest-arg -m --pytest-arg smoke,fast
trace MSYS_NO_PATHCONV=1 $CLI pipelines test my_pipeline tests tests/test_orders.py::test_backfill /Workspace/Users/someone@example.com/tests/test_other.py --pytest-arg=-k --pytest-arg='orders and not slow' --pytest-arg -m --pytest-arg smoke,fast --no-wait
trace print_requests.py //updates

title "Relative paths resolve from the working directory"
cd tests
trace $CLI pipelines test my_pipeline test_orders.py::test_valid_order
trace $CLI pipelines test my_pipeline test_orders.py::test_valid_order --no-wait
cd ..
trace print_requests.py //updates

Expand Down
2 changes: 1 addition & 1 deletion acceptance/pipelines/test/submit-error/output.txt
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@

=== A submit error from the service is shown as is, and no update ID is printed
>>> musterr [CLI] pipelines test --pipeline-id [UUID][0]
Error: An active update '[UUID][1]' already exists for pipeline '[UUID][0]'.
Error: An active update '[UUID][1]' already exists for pipeline '[UUID][2]'.

>>> print_requests.py //updates
{
Expand Down
5 changes: 4 additions & 1 deletion acceptance/pipelines/test/submit-error/script
Original file line number Diff line number Diff line change
@@ -1,3 +1,6 @@
pipeline_id=$($CLI pipelines create --json '{"name": "orders", "root_path": "/Workspace/Users/someone@example.com/orders"}' | jq -r .pipeline_id)
rm "$OUT_REQUESTS"

title "A submit error from the service is shown as is, and no update ID is printed"
trace musterr $CLI pipelines test --pipeline-id 11111111-2222-3333-4444-555555555555
trace musterr $CLI pipelines test --pipeline-id $pipeline_id
trace print_requests.py //updates
Loading
Loading