Repository navigation
Build a JUnit XML report from the results of a test run - #7299
tharunbhuk-db wants to merge 1 commit into
Conversation
cf934e0 to
cc2e61a
Compare
2c84257 to
8542365
Compare
|
Commit: Regression test reportSelected 13 changed tests (limit=20, 0 not selected) 13 tests: regression=13 unreleased=0 coverage=0 failing=0
TestAccept/pipelines/databricks-cli-help: branch ✅ | main (688c228) ❌ | latest (v1.25.0) ❌main (688c228): latest (v1.25.0): same output as main (688c228) TestAccept/pipelines/test/arguments: branch ✅ | main (688c228) ❌ | latest (v1.25.0) ❌main (688c228): latest (v1.25.0): same output as main (688c228) TestAccept/pipelines/test/immutable: branch ✅ | main (688c228) ❌ | latest (v1.25.0) ❌main (688c228): latest (v1.25.0): same output as main (688c228) TestAccept/pipelines/test/pipeline-id: branch ✅ | main (688c228) ❌ | latest (v1.25.0) ❌main (688c228): latest (v1.25.0): same output as main (688c228) TestAccept/pipelines/test/results/failed: branch ✅ | main (688c228) ❌ | latest (v1.25.0) ❌main (688c228): latest (v1.25.0): same output as main (688c228) TestAccept/pipelines/test/results/incomplete: branch ✅ | main (688c228) ❌ | latest (v1.25.0) ❌main (688c228): latest (v1.25.0): same output as main (688c228) TestAccept/pipelines/test/results/passed: branch ✅ | main (688c228) ❌ | latest (v1.25.0) ❌main (688c228): latest (v1.25.0): same output as main (688c228) TestAccept/pipelines/test/run-control/canceled: branch ✅ | main (688c228) ❌ | latest (v1.25.0) ❌main (688c228): latest (v1.25.0): same output as main (688c228) TestAccept/pipelines/test/run-control/not-test-update: branch ✅ | main (688c228) ❌ | latest (v1.25.0) ❌main (688c228): latest (v1.25.0): same output as main (688c228) TestAccept/pipelines/test/run-control/resume: branch ✅ | main (688c228) ❌ | latest (v1.25.0) ❌main (688c228): latest (v1.25.0): same output as main (688c228) TestAccept/pipelines/test/run-control/timeout: branch ✅ | main (688c228) ❌ | latest (v1.25.0) ❌main (688c228): latest (v1.25.0): same output as main (688c228) TestAccept/pipelines/test/selection: branch ✅ | main (688c228) ❌ | latest (v1.25.0) ❌main (688c228): latest (v1.25.0): same output as main (688c228) TestAccept/pipelines/test/submit-error: branch ✅ | main (688c228) ❌ | latest (v1.25.0) ❌main (688c228): latest (v1.25.0): same output as main (688c228) |
✅ Integration test reportCommit: 744d6ca
Top 3 slowest tests (at least 2 minutes):
|
cb0a526 to
025649f
Compare
b4df318 to
7c61f68
Compare
025649f to
23e0bb0
Compare
| } | ||
| data = append([]byte(xml.Header), data...) | ||
| data = append(data, '\n') | ||
| if err := os.WriteFile(path, data, 0o644); err != nil { |
There was a problem hiding this comment.
os.WriteFile truncates first, so a crash mid-write leaves a half-written file that CI reads as broken. Could we use atomicfile.Write from libs/atomicfile? bundle/deploy/state_pull.go already uses it.
There was a problem hiding this comment.
Done. The report is written with atomicfile.Write.
| // A parameter ID in brackets, such as test_cast[a::int], stays with the test name. | ||
| func junitCaseName(testID string) (string, string) { | ||
| params := "" | ||
| if i := strings.Index(testID, "["); i >= 0 { |
There was a problem hiding this comment.
strings.Index(testID, "[") searches the whole ID, so tests/[weird]/test_a.py::test_x gets classname "" and name tests.[weird].test_a.py::test_x. Searching only after the first :: fixes it. Worth adding that case to TestJUnitCaseName.
There was a problem hiding this comment.
Done. junitCaseName looks for [ only after the first ::, and TestJUnitCaseName has the tests/[weird]/test_a.py::test_x case.
|
|
||
| if problem != "" { | ||
| suite.Cases = append(suite.Cases, junitTestCase{ | ||
| Name: "update " + updateID, |
There was a problem hiding this comment.
This entry is named update <id>, while pre-update errors are named pipelines test (line 167). On the timeout path, the update is still running and the message doesn't start with RUN INCOMPLETE:. Could we use one naming scheme, with a clear message for the local-timeout case?
There was a problem hiding this comment.
The timeout entry now says "stopped waiting for update X after 10m; the update keeps running, to get its results run: …" (#7434). One naming scheme for the entries: I'll do that in a follow-up.
| // Every exit writes the report, so that an older report never survives. | ||
| report := &testReport{path: junitPath, out: cmd.OutOrStdout(), suiteName: pipelineID} | ||
| if report.suiteName == "" && len(args) > 0 { | ||
| report.suiteName = args[0] |
There was a problem hiding this comment.
Outside a project, positional args are selections, so pipelines test tests/test_x.py --junit-xml r.xml names the suite tests/test_x.py, and with no args the suite name is empty. Could we use the pipeline name or ID once it resolves?
There was a problem hiding this comment.
Done in #7434. Once the pipeline resolves, the suite name is the KEY, or the pipeline ID with --pipeline-id.
|
|
||
| --junit-xml writes one entry per test with its outcome and duration. An | ||
| incomplete run, a run without tests, or a timeout adds a failing entry that | ||
| names the cause. With workspace.sync_mode immutable, test names keep the path |
There was a problem hiding this comment.
Non-blocking: thanks for documenting this. Classnames that change with each deploy break CI test-history views (flaky detection, trends). A follow-up could name cases relative to the snapshot root.
There was a problem hiding this comment.
I'll do that in a follow-up.
| trace musterr $CLI pipelines test --pipeline-id 1234 --no-wait --junit-xml out.no-wait.xml | ||
|
|
||
| title "An invalid flag replaces an older report" | ||
| echo '<testsuites><testsuite name="1234" tests="1"><testcase name="test_old"></testcase></testsuite></testsuites>' > out.flag-error.xml |
There was a problem hiding this comment.
Non-blocking: the "an older report is replaced" check now appears in five scripts: arguments, submit-error, run-control, outside, and an invalid project. Each shows the same thing, that a failure before the update still rewrites the report. One case before the request and one at submit would cover it, and the other goldens could go.
There was a problem hiding this comment.
Done. Only arguments (before the request) and submit-error (at submit) check it now.
cd64420 to
17f12d7
Compare
744d6ca to
58df287
Compare
17f12d7 to
2f8b6d9
Compare
2f8b6d9 to
e225070
Compare
58df287 to
9d83e17
Compare
e225070 to
1059c51
Compare
9d83e17 to
062047b
Compare
The report has one testcase per collected test case, named as pytest's JUnit report names it, with its outcome and duration. A case without a final outcome is an error, and a run problem, such as an incomplete run, adds a failing entry. A bracket in a folder name stays part of the path, and the file is replaced atomically so that a reader never sees a partly written report. The command does not write the report yet. Co-authored-by: Isaac <no-reply@databricks.com>
1059c51 to
66654a5
Compare
062047b to
c7628c9
Compare
## 🥞 Stacked PR Use this [link](https://github.com/databricks/cli/pull/7297/files) to review incremental changes. - [**stack/pipelines-test-pr1-command**](databricks#7297) [[Files changed](https://github.com/databricks/cli/pull/7297/files)] ← _this PR_ - [stack/pipelines-test-bundle-key](databricks#7431) [[Files changed](https://github.com/databricks/cli/pull/7431/files)] - [stack/pipelines-test-read-events](databricks#7432) [[Files changed](https://github.com/databricks/cli/pull/7432/files)] - [stack/pipelines-test-print-results](databricks#7433) [[Files changed](https://github.com/databricks/cli/pull/7433/files)] - [stack/pipelines-test-pr2-output](databricks#7298) [[Files changed](https://github.com/databricks/cli/pull/7298/files)] - [stack/pipelines-test-a4-run-control](databricks#7426) [[Files changed](https://github.com/databricks/cli/pull/7426/files)] - [stack/pipelines-test-pr3-junit](databricks#7299) [[Files changed](https://github.com/databricks/cli/pull/7299/files)] - [stack/pipelines-test-junit-flag](databricks#7434) [[Files changed](https://github.com/databricks/cli/pull/7434/files)] --------- - Add `databricks pipelines test --pipeline-id <id> [TEST_SELECTION...]`: starts a test-only update and prints its IDs. - Experimental: runs only with `DATABRICKS_CLI_PIPELINES_TEST=true`. - Selections are workspace paths or paths relative to `root_path`, sent as typed. `--pytest-arg` accepts only `-k` and `-m`. Tests: unit tests; acceptance in `acceptance/pipelines/test/{arguments,pipeline-id,submit-error}`. This pull request and its description were written by Isaac. Co-authored-by: Isaac <no-reply@databricks.com>
## 🥞 Stacked PR - [stack/pipelines-test-pr1-command](databricks#7297) [[Files changed](https://github.com/databricks/cli/pull/7297/files)] - [**stack/pipelines-test-bundle-key**](databricks#7431) [[Files changed](https://github.com/databricks/cli/pull/7431/files)] ← _this PR_ - [stack/pipelines-test-read-events](databricks#7432) [[Files changed](https://github.com/databricks/cli/pull/7432/files)] - [stack/pipelines-test-print-results](databricks#7433) [[Files changed](https://github.com/databricks/cli/pull/7433/files)] - [stack/pipelines-test-pr2-output](databricks#7298) [[Files changed](https://github.com/databricks/cli/pull/7298/files)] - [stack/pipelines-test-a4-run-control](databricks#7426) [[Files changed](https://github.com/databricks/cli/pull/7426/files)] - [stack/pipelines-test-pr3-junit](databricks#7299) [[Files changed](https://github.com/databricks/cli/pull/7299/files)] - [stack/pipelines-test-junit-flag](databricks#7434) [[Files changed](https://github.com/databricks/cli/pull/7434/files)] --------- - In a bundle, identify the pipeline by KEY: `pipelines test [flags] [KEY] [TEST_SELECTION...]`. - Relative selections are local paths, mapped to their deployed paths. - A first argument that is not a KEY gets an error that names the bundle's pipelines and the working command. Tests: unit tests; acceptance in `acceptance/pipelines/test/{selection,immutable}`. This pull request and its description were written by Isaac. Co-authored-by: Isaac <no-reply@databricks.com>
🥞 Stacked PR
Tests: unit tests.
This pull request and its description were written by Isaac.