Amogan/daq release issue568

[andrewmogan]

Description

This PR addresses daq-release #568. Our nightly integration test workflows categorize our integration tests into "core" and "extended" tests. Core tests are a selection of daqsystemtest tests and the listrev_test from listrev. Extended tests are all non-core tests. This distinction doesn't exist in dbt-build --integtest, and getting the right tests to run is currently done in a somewhat hacky way.

To make it easier to select integration tests for an arbitrary release, this PR adds functionality to dunedaq_integtest_bundle.sh to define "suites" of tests, allowing it to effectively replace dbt-build --integtest. Test suites can be defined in a text file in daqsystemtest/scripts/test_suites/. Each line in this file should be formatted as <repo_name>:<test_name>; see core.txt for an example. Assuming a corresponding suite file exists, you run this as dunedaq_integtest_bundle.sh -s <my_suite> [etc.]. As of now, the only valid suite is "core," where "extended" is a special case treated as the inverse of "core." But this functionality allows for defining any arbitrary set of tests as a named suite, which some developers may find useful. Even if no suites are ever added, this gives us a single source of truth for our list of core tests.

In order for this to be used as part of our nightly core integration test workflow, I also added a --junit-xml option which appends --junit-xml <repo_name>_<test_name>_results.xml to PYTEST_COMMAND if supplied. Our nightly core integration test workflow uses these junit XML files to build the integration test page on the CI dashboard.

Incidentally, this fixes an issue where some daqsystemtest tests that were not defined as core tests weren't running as part of any nightly integration test workflow.

I've provided usage examples below. Note that passing -k, -x, -r, or -R in combination with -s causes the script to emit a warning, but does not stop the tests from running. This is because a) the additional arguments don't always behave as one might intuitively expect, and b) these arguments defeat the purpose of defining test suites in the first place. This could arguably be an error instead of a warning, but I've kept it as a warning for now.

dunedaq_integtest_bundle.sh -s core
dunedaq_integtest_bundle.sh -s extended --junit-xml
dunedaq_integtest_bundle.sh -s core --junit-xml --pytest-options "--dunerc-option log-level debug"

Type of change

• Documentation (non-breaking change that adds or improves the documentation)
• New feature or enhancement (non-breaking change which adds functionality)
• Optimization (non-breaking change that improves code/performance)
• Bug fix (non-breaking change which fixes an issue)
• Breaking change (whatever its nature)

Testing checklist

In progress. I've noticed that not all the correct tests are listed in some corner cases, hence the draft status.

• Full set of integration tests pass (dunedaq_integtest_bundle.sh)

Further checks

• Code is commented where needed, particularly in hard-to-understand areas