Repository navigation
Wait for and print the results of databricks pipelines test - #7298
tharunbhuk-db wants to merge 1 commit into
Conversation
2c84257 to
8542365
Compare
|
Commit: Regression test reportSelected 9 changed tests (limit=20, 0 not selected) 9 tests: regression=9 unreleased=0 coverage=0 failing=0
TestAccept/pipelines/databricks-cli-help: branch ✅ | main (688c228) ❌ | latest (v1.25.0) ❌main (688c228): latest (v1.25.0): same output as main (688c228) TestAccept/pipelines/test/arguments: branch ✅ | main (688c228) ❌ | latest (v1.25.0) ❌main (688c228): latest (v1.25.0): same output as main (688c228) TestAccept/pipelines/test/immutable: branch ✅ | main (688c228) ❌ | latest (v1.25.0) ❌main (688c228): latest (v1.25.0): same output as main (688c228) TestAccept/pipelines/test/pipeline-id: branch ✅ | main (688c228) ❌ | latest (v1.25.0) ❌main (688c228): latest (v1.25.0): same output as main (688c228) TestAccept/pipelines/test/results/failed: branch ✅ | main (688c228) ❌ | latest (v1.25.0) ❌main (688c228): latest (v1.25.0): same output as main (688c228) TestAccept/pipelines/test/results/incomplete: branch ✅ | main (688c228) ❌ | latest (v1.25.0) ❌main (688c228): latest (v1.25.0): same output as main (688c228) TestAccept/pipelines/test/results/passed: branch ✅ | main (688c228) ❌ | latest (v1.25.0) ❌main (688c228): latest (v1.25.0): same output as main (688c228) TestAccept/pipelines/test/selection: branch ✅ | main (688c228) ❌ | latest (v1.25.0) ❌main (688c228): latest (v1.25.0): same output as main (688c228) TestAccept/pipelines/test/submit-error: branch ✅ | main (688c228) ❌ | latest (v1.25.0) ❌main (688c228): latest (v1.25.0): same output as main (688c228) |
✅ Integration test reportCommit: b45969f
Top 3 slowest tests (at least 2 minutes):
|
8542365 to
cb0a526
Compare
025649f to
23e0bb0
Compare
| if incompleteReason == "" { | ||
| summary := run.summaries[0] | ||
| testsFailed := summary.Total == 0 || summary.Failed > 0 || summary.Errors > 0 | ||
| if !testsFailed && state == pipelines.UpdateInfoStateCompleted { |
There was a problem hiding this comment.
A COMPLETED update whose tests all passed returns nil here without checking run.fatalError, so an update that also reported a fatal error exits 0. Could we add run.fatalError == "" to this condition, with a TestTestRunError case for it?
Related: a run where every test is SKIPPED also returns nil here. That matches the design (an all-skipped run completes), but CI and agents will read it as green, so the help should say so explicitly.
There was a problem hiding this comment.
Done. testRunError also checks run.fatalError == "", with a test case. The help says an all-skipped run exits 0.
| UpdateId: tw.updateID, | ||
| }) | ||
| if err != nil { | ||
| return "", err |
There was a problem hiding this comment.
A GetUpdate error (and the same for the events read just below) ends the wait with the raw error and no --resume hint, even though the update keeps running. The SDK client already retries transient HTTP errors, so no extra retry logic is needed here. Could the error just include the same resume command as the timeout path?
There was a problem hiding this comment.
I'll do this in a follow-up.
| updateID := resumeUpdateID | ||
| if !resume { | ||
| req := newStartTestUpdateRequest(target.selections, pytestArgs) | ||
| updateID, err = startTestUpdate(ctx, target.w, target.pipelineID, req) |
There was a problem hiding this comment.
This POST uses the signal context, so a Ctrl+C during Start Update loses the update ID, and the update runs with no way to find it from this output. Could the POST use cmd.Context()?
Also on this line: when the service rejects the start because an update is already active, a short static hint after its message would give agents the next step, for example "Run 'databricks pipelines stop KEY' to cancel it, or wait for it to finish." There's no need to parse the ID out of the message.
There was a problem hiding this comment.
Done in #7426: Start Update uses cmd.Context(), tested by TestInterruptDuringStartUpdateKeepsTheUpdateID. The already-active hint: I'll do that in a follow-up.
| fmt.Fprintln(out) | ||
|
|
||
| if incompleteReason != "" { | ||
| fmt.Fprintf(out, "RUN INCOMPLETE: %s.\n", incompleteReason) |
There was a problem hiding this comment.
The most likely cause of INCOMPLETE is events still arriving after the terminal state, and the update can still be read later. Could we print the --resume command here too?
There was a problem hiding this comment.
I'll do this in a follow-up.
|
|
||
| // testVerdictErrorPrefix starts the message of the error by which the service reports that tests | ||
| // failed, such as "[PIPELINE_TEST_FAILED] 1 of the pipeline's tests failed (1 passed, 0 skipped).". | ||
| const testVerdictErrorPrefix = "[PIPELINE_TEST_FAILED]" |
There was a problem hiding this comment.
Recognizing the service verdict by message prefix is fragile. The exit code doesn't depend on it, only the output does. The SDK's ErrorDetail has no error class yet, so a comment saying this is temporary until a class is exposed would help.
There was a problem hiding this comment.
Done in #7432. A comment says the message is matched because ErrorDetail has no error class yet.
| // update ended in another way or reports an error of its own. The service's own verdict, | ||
| // such as "[PIPELINE_TEST_FAILED] 1 of the pipeline's tests failed", is not. | ||
| ended := state == pipelines.UpdateInfoStateFailed || state == pipelines.UpdateInfoStateCompleted | ||
| if testsFailed && ended && run.fatalError == "" { |
There was a problem hiding this comment.
Every failure path exits 1. Test failures and zero-collected runs print nothing to stderr, while infra failures print Error:, so an agent tells "fix the code" from "retry" only from the output. pipelines run behaves the same, so distinct exit codes would be a CLI-wide decision. For this stack, could the help just say what exit 0 and 1 mean? Small related nit: with --junit-xml, the JUnit report: line prints after the summary, so tail -1 misses the summary; printing it before the summary would keep the summary last.
There was a problem hiding this comment.
Done. The help says what exit 0 and 1 mean. The JUnit report: line prints before the summary from #7434.
| summary := run.summaries[0] | ||
| switch summary.Total { | ||
| case 0: | ||
| fmt.Fprintln(out, "No tests were collected.") |
There was a problem hiding this comment.
Agents hit this often, and the message gives no hint. Naming what was run would make it self-correcting, e.g. No tests were collected from <testfile>. Check the test selection and any --pytest-arg filters. (I'd avoid restating pytest's naming rules here, since projects can change them in their pytest configuration.) But if we can give a bit better message that would be nice.
There was a problem hiding this comment.
I'll do this in a follow-up.
| testCasePassed testCaseStatus = "PASSED" | ||
| testCaseFailed testCaseStatus = "FAILED" | ||
| testCaseSkipped testCaseStatus = "SKIPPED" | ||
| testCaseError testCaseStatus = "ERROR" |
There was a problem hiding this comment.
should this be errored? I see we do error and errored at some places would be good if we can do consistent.
There was a problem hiding this comment.
These follow the proto: the case status is ERROR and the summary field is errored. The summary line uses the status names (ERROR=).
| @@ -0,0 +1,799 @@ | |||
| # 200 cases emit 601 events, more than two pages of List Pipeline Events. | |||
There was a problem hiding this comment.
Non-blocking: instead of committing 800 lines of identical tests, could the script generate this file before pipelines deploy? The fake collects tests with a regex over the deployed files (libs/testserver/pipelines_testing.go:106), so this works the same and runs on bash 3.2 and Git Bash:
for i in $(seq 0 199); do printf 'def test_case_%03d():\n assert True\n\n' "$i"; done > tests/test_many.pyAlso worth checking how many cases you actually need. The CLI doesn't send max_results, so the fake pages at its default of 25 events (pipelines_testing.go:332), and the output shows 8 pages read. Fewer cases may cover the same paths. The page math reads better in the script than as a comment in a generated file.
There was a problem hiding this comment.
Done. The unit test TestReadTestEvents200Cases in #7432 replaces it. The reader sends max_results=250.
| assert.Equal(t, 400, response.StatusCode) | ||
| } | ||
|
|
||
| func TestPipelineStartUpdateRejectsInvalidTestOptions(t *testing.T) { |
There was a problem hiding this comment.
Non-blocking: this file tests the fake more deeply than the CLI contract needs, and a few could go:
TestPipelineStartUpdateRejectsInvalidTestOptionsandTestPipelineStartUpdateRejectsTestOptionsWithoutTestOnly, together with the fake'spipelineTestOptionsError. The CLI validates--pytest-argand makes selections absolute before any request, and it always sendstest_only: true, sopipelines testnever reaches this validation.TestPipelineFiniteUpdateIsActiveUntilItEnds,TestPipelineUpdateThatIsNeverPolledEndsWithItsFirstStateandTestPipelineStopHidesEventsOfStatesNotReached. They pin the fake's state timing, and what the CLI sees (409, CANCELED) is golden inrun-controlandsubmit-error.TestPipelineEventsPages. Paging is covered by the CLI's own reader test andmany-cases.
TestCollectPipelineTests, TestPipelineTestUpdate and TestPipelineTestUpdateConflictAndStop cover what the acceptance tests rely on. Also, line 242 repeats the assertion on line 241.
There was a problem hiding this comment.
Done. The stack no longer changes libs/testserver.
66e7f7e to
62049a1
Compare
23e0bb0 to
b45969f
Compare
b45969f to
e8a772a
Compare
8329c9f to
2036088
Compare
e8a772a to
e7de143
Compare
|
Follow-ups from review:
|
The command waits for the test-only update, prints each test case as it finishes, then the failure details and a summary line. It exits with code 0 only for a complete run where at least one test ran and none failed or errored, and a completed update that reports a fatal error does not pass. Test IDs are shown relative to the project folder. With the no-wait flag, the command returns after the update starts. The wait takes the steps of the bundle pipeline runner's wait loop. The acceptance tests stub the service per case, as the other pipelines tests do. Co-authored-by: Isaac <no-reply@databricks.com>
2036088 to
42a9409
Compare
e7de143 to
d0d7138
Compare
## 🥞 Stacked PR Use this [link](https://github.com/databricks/cli/pull/7297/files) to review incremental changes. - [**stack/pipelines-test-pr1-command**](databricks#7297) [[Files changed](https://github.com/databricks/cli/pull/7297/files)] ← _this PR_ - [stack/pipelines-test-bundle-key](databricks#7431) [[Files changed](https://github.com/databricks/cli/pull/7431/files)] - [stack/pipelines-test-read-events](databricks#7432) [[Files changed](https://github.com/databricks/cli/pull/7432/files)] - [stack/pipelines-test-print-results](databricks#7433) [[Files changed](https://github.com/databricks/cli/pull/7433/files)] - [stack/pipelines-test-pr2-output](databricks#7298) [[Files changed](https://github.com/databricks/cli/pull/7298/files)] - [stack/pipelines-test-a4-run-control](databricks#7426) [[Files changed](https://github.com/databricks/cli/pull/7426/files)] - [stack/pipelines-test-pr3-junit](databricks#7299) [[Files changed](https://github.com/databricks/cli/pull/7299/files)] - [stack/pipelines-test-junit-flag](databricks#7434) [[Files changed](https://github.com/databricks/cli/pull/7434/files)] --------- - Add `databricks pipelines test --pipeline-id <id> [TEST_SELECTION...]`: starts a test-only update and prints its IDs. - Experimental: runs only with `DATABRICKS_CLI_PIPELINES_TEST=true`. - Selections are workspace paths or paths relative to `root_path`, sent as typed. `--pytest-arg` accepts only `-k` and `-m`. Tests: unit tests; acceptance in `acceptance/pipelines/test/{arguments,pipeline-id,submit-error}`. This pull request and its description were written by Isaac. Co-authored-by: Isaac <no-reply@databricks.com>
## 🥞 Stacked PR - [stack/pipelines-test-pr1-command](databricks#7297) [[Files changed](https://github.com/databricks/cli/pull/7297/files)] - [**stack/pipelines-test-bundle-key**](databricks#7431) [[Files changed](https://github.com/databricks/cli/pull/7431/files)] ← _this PR_ - [stack/pipelines-test-read-events](databricks#7432) [[Files changed](https://github.com/databricks/cli/pull/7432/files)] - [stack/pipelines-test-print-results](databricks#7433) [[Files changed](https://github.com/databricks/cli/pull/7433/files)] - [stack/pipelines-test-pr2-output](databricks#7298) [[Files changed](https://github.com/databricks/cli/pull/7298/files)] - [stack/pipelines-test-a4-run-control](databricks#7426) [[Files changed](https://github.com/databricks/cli/pull/7426/files)] - [stack/pipelines-test-pr3-junit](databricks#7299) [[Files changed](https://github.com/databricks/cli/pull/7299/files)] - [stack/pipelines-test-junit-flag](databricks#7434) [[Files changed](https://github.com/databricks/cli/pull/7434/files)] --------- - In a bundle, identify the pipeline by KEY: `pipelines test [flags] [KEY] [TEST_SELECTION...]`. - Relative selections are local paths, mapped to their deployed paths. - A first argument that is not a KEY gets an error that names the bundle's pipelines and the working command. Tests: unit tests; acceptance in `acceptance/pipelines/test/{selection,immutable}`. This pull request and its description were written by Isaac. Co-authored-by: Isaac <no-reply@databricks.com>
🥞 Stacked PR
--no-waitreturns after the update starts.Tests: unit tests; acceptance in
acceptance/pipelines/test/results/*, with the service stubbed per case.This pull request and its description were written by Isaac.