Skip to content

testmask: run tests named in the last commit message - #7524

Merged
denik merged 2 commits into
mainfrom
testmask-commit-names
Nov 12, 2026
Merged

denik merged 2 commits into
mainfrom
testmask-commit-names

Conversation

@denik

@denik denik commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Changes

PR runs only cover the tests a branch changed. With this PR, a test can also be requested by naming it in the last commit message of the PR:

empty commit to run tests

aws: TestAccept/bundle/resources/jobs
cloud: TestAccept/bundle/resources/pipelines
  • aws:, azure:, gcp: select the test on that cloud only, cloud: on every cloud. The prefix is lowercase and must start the line (surrounding whitespace is ignored). A line can list several whitespace-separated names. Lines with other prefixes are ignored, and so is any line with a word that does not start with Test, so prose such as aws: fix flaky TestAccept retries cannot select the whole suite.
  • Local (testserver) runs pick up every named test, whatever its prefix.
  • TestAccept/<path> selects every test dir equal to or under <path>/ (path-segment prefix, so jobs does not match jobsx), with all variants. TestAccept/<dir>/KEY=value[/KEY=value...] selects only the matching variants, as go test names them. Each KEY=value must appear in one of the dir's variants (per out.test.toml), so a misspelled key or value is reported instead of running every variant or none. A bare TestAccept selects everything. Names that match no acceptance test get a warning in the test log and are otherwise ignored.
  • Named tests are added on top of the DATABRICKS_TEST_SELECT_CHANGED selection (PR cloud run, ./task local-select) and the windows/macOS PR subset (TESTS_SELECT_SUBSET_PCT), and they don't count toward those limits. Full runs (Linux PR cell, nightly) are unchanged.
  • On cloud, a named test still needs Cloud = true and a matching CloudEnvs. It does run even if it is CloudSlow.
  • tools/testmask (only for pull_request events, via GITHUB_EVENT_NAME) adds the test target when the head commit has such a line, so the test jobs and the cloud trigger run even for a PR whose files would not trigger them. The merge queue ignores the message. Failing to read the message only logs a warning.

Where it lives:

  • acceptance/internal/selection/commit.go: parser, name resolution and reading the last commit message. On the Merge <head> into <base> commit GitHub creates for refs/pull/N/merge (which both the PR test job and the cloud run check out), it reads the PR head commit's message instead.
  • acceptance/acceptance_test.go, acceptance/subset_test.go: wiring.
  • tools/testmask: trigger detection (a small regex that mirrors the parser; the tools module cannot import acceptance/internal).
  • Documented in acceptance/README.md.

Why

You can then run specific tests on specific clouds from a PR (for example, to check a fix for a cloud-only failure) by pushing an empty commit, without editing test files or running the full cloud suite.

Decisions:

  • The cloud side needs no change. cli-isolated-pr.yml runs this repo's cloud-select task (alias integration-short-skiplocal) from the PR checkout with CLOUD_ENV set, so the selection code at the PR's commit already knows which cloud it is on (getCloudEnvBase: aws/ucws -> aws, gcp/gcp-ucws -> gcp, azure -> azure).
  • Prefix matching follows the path-segment rule above, because the example in the request (TestAccept/bundle/resources/jobs) is a directory of many tests.
  • An unmatched name only warns and does not fail the run, so a typo in a commit message cannot block CI.
  • Review points left as is: the CloudSlow bypass is per dir, so naming one variant lifts it for the whole dir (the cloud run still filters variants). A local git merge main commit at HEAD hides the names locally; CI is unaffected because it reads the PR head commit. testmask repeats the prefix list because the tools module cannot import acceptance/internal.
  • Explicitly named tests bypass CloudSlow because someone asked for them by name. They do not bypass Cloud = false: such tests cannot run on cloud.

Tests

  • Unit tests for the parser (prefixes, unknown and uppercase prefixes ignored, mid-line mentions ignored, multiple lines and names, trailing whitespace and CRLF, no matches), cloud filtering, name resolution (prefix, sibling with the same prefix, variants, unmatched), variant matching, and reading the message on a plain commit, a GitHub PR merge commit and an ordinary merge commit. Also unit tests for testmask's detection.
  • Local: DATABRICKS_TEST_SELECT_CHANGED=50 go test ./acceptance -run '^TestAccept$' with a commit naming aws: TestAccept/cmd/fs/cp/file-to-file, cloud: TestAccept/bundle/validate/enum/DMS=true and gcp: TestAccept/cmd/fs/cp/dir-to-dir ran all three, and for enum only the DMS=true variant.
  • AWS (testme-aws -short, same commit): selected file-to-file (passed) and enum (skipped, Cloud = false). The gcp-only name was not selected.
  • The last commit of this PR names aws: TestAccept/cmd/fs/cp/file-to-file, so this PR's own cloud run exercises the feature.

This pull request and its description were written by Isaac.

Lines like "aws: TestAccept/<path>" or "cloud: TestAccept/<path>" in the
last commit message of a PR add those tests to the PR cloud run on the
named clouds and to the local selection. See acceptance/README.md.

aws: TestAccept/cmd/fs/cp/file-to-file

Co-authored-by: Isaac <no-reply@databricks.com>
@denik denik added the ai-auto PRs made by AI loops & factories, label Nov 11, 2026
…sk trigger

aws: TestAccept/cmd/fs/cp/file-to-file

Co-authored-by: Isaac <no-reply@databricks.com>
@eng-dev-ecosystem-bot

Copy link
Copy Markdown
Collaborator

✅ Integration test report

Commit: ad0826a

Run: 43123713025

Env 🤯​MISS 💚​RECOVERED ✅​pass 🙈​skip Time
💚​ aws linux-2core-8gb 1 271 15 6:39
💚​ aws-windows-latest-4core-16gb 1 273 13 4:19
🤯​ azure linux-2core-8gb 1 1 269 15 6:38
🤯​ azure-windows-latest-4core-16gb 1 1 271 13 4:51
🤯​ gcp linux-2core-8gb 1 1 270 15 7:55
🤯​ gcp-windows-latest-4core-16gb 1 1 272 13 4:34
Test Name aws linux-2core-8gb aws-windows-latest-4core-16gb azure linux-2core-8gb azure-windows-latest-4core-16gb gcp linux-2core-8gb gcp-windows-latest-4core-16gb
💚​ TestAccept 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
🤯​ TestAccept/cmd/fs/cp/file-to-file ✅​p ✅​p 🤯​M 🤯​M 🤯​M 🤯​M
Top 6 slowest tests (at least 2 minutes):
duration env testname
4:04 azure-windows-latest-4core-16gb TestAccept
4:01 gcp linux-2core-8gb TestAccept
3:58 aws linux-2core-8gb TestAccept
3:52 azure linux-2core-8gb TestAccept
3:46 gcp-windows-latest-4core-16gb TestAccept
3:45 aws-windows-latest-4core-16gb TestAccept

@denik
denik marked this pull request as ready for review November 12, 2026 00:23
@denik
denik requested a review from a team as a code owner November 12, 2026 00:23
@denik
denik requested a review from andrewnester November 12, 2026 00:23
@denik
denik enabled auto-merge November 12, 2026 00:23

@andrewnester andrewnester left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The selection cmd (acceptance/internal/selection/cmd) still prints only the changed tests, so there is no local way to see what a commit message selects without running the suite. Worth adding the named tests there, or a follow-up?


// CloudSlow only narrows an already-enabled cloud run: skip it under -short.
if isTruePtr(config.CloudSlow) && testing.Short() {
if isTruePtr(config.CloudSlow) && testing.Short() && !named {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A prefix name like cloud: TestAccept lifts CloudSlow for every dir under it, more than the -short nightly runs and inside cloud-select's 2h timeout. Should the bypass need an exact dir or variant name?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. Fixed in #7548: only a name that is the dir itself or one of its variants lifts CloudSlow (new Named.Exact). A prefix such as cloud: TestAccept or TestAccept/bundle still selects the subtree, but its CloudSlow tests stay skipped under -short.

— AI agent (harness_auto, opus5.5, isaac_claude)

continue
}
words := strings.Fields(m[2])
if len(words) == 0 || slices.ContainsFunc(words, func(word string) bool { return !strings.HasPrefix(word, "Test") }) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A line like aws: `TestAccept/x` (backticks, or a trailing word) is dropped as prose with no warning, unlike an unmatched name. Can those lines be logged too, so a silent no-op is visible?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. Fixed in #7548: a cloud-prefixed line that is not read as names (backticks, a trailing word, prose) now logs WARNING: ignoring "<line>" ... next to the unmatched-name warning, and the selection cmd prints it too.

— AI agent (harness_auto, opus5.5, isaac_claude)

}
t.Chdir(t.TempDir())

git := func(args ...string) string {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

runSelectionGit in selection_test.go already runs git with a fixed identity for this package's tests. Can this reuse it instead of a second git closure plus config calls?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. Done in #7548: runSelectionGit now returns git's output and TestLastCommitMessage uses it; commit_test.go moved to package selection_test to share it.

— AI agent (harness_auto, opus5.5, isaac_claude)

@denik
denik added this pull request to the merge queue Nov 12, 2026
Merged via the queue into main with commit 69de469 Nov 12, 2026
27 checks passed
@denik
denik deleted the testmask-commit-names branch November 12, 2026 07:43
@eng-dev-ecosystem-bot

Copy link
Copy Markdown
Collaborator

❌ Integration test report

Commit: 69de469

Run: 43177480887

Env ❌​FAIL 🟨​KNOWN 💚​RECOVERED ✅​pass 🙈​skip Time
❌​ aws linux-2core-8gb 39 3 9 1511 1279 155:39
❌​ aws-windows-latest-4core-16gb 39 1 11 1430 1306 188:18
❌​ azure linux-2core-8gb 41 1 11 1489 1287 130:39
❌​ azure-windows-latest-4core-16gb 39 1 11 1410 1314 157:06
❌​ gcp linux-2core-8gb 39 1 11 1498 1285 146:35
❌​ gcp-windows-latest-4core-16gb 41 1 11 1415 1312 182:12
55 interesting tests: 43 FAIL, 9 RECOVERED, 3 KNOWN
Test Name aws linux-2core-8gb aws-windows-latest-4core-16gb azure linux-2core-8gb azure-windows-latest-4core-16gb gcp linux-2core-8gb gcp-windows-latest-4core-16gb
🟨​ TestAccept 🟨​K 🟨​K 🟨​K 🟨​K 🟨​K 🟨​K
💚​ TestAccept/bundle/resources/apps/lifecycle-started 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/apps/lifecycle-started-omitted 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/apps/lifecycle-started-omitted/DMS=true 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/apps/lifecycle-started-toggle 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/apps/lifecycle-started-toggle/DMS= 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/apps/lifecycle-started-toggle/DMS=true 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/apps/lifecycle-started/DMS= 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/apps/lifecycle-started/DMS=true 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
❌​ TestAccept/bundle/resources/postgres_branches/update_protected ❌​F ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/resources/postgres_branches/update_protected/DMS= ❌​F ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/resources/postgres_branches/update_protected/DMS=true ❌​F ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/resources/postgres_branches/update_ttl ❌​F ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/resources/postgres_branches/update_ttl/DMS= ❌​F ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/resources/postgres_branches/update_ttl/DMS=true ❌​F ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/resources/postgres_databases/update ❌​F ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/resources/postgres_databases/update/DMS= ❌​F ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/resources/postgres_databases/update/DMS=true ❌​F ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/resources/postgres_endpoints/remove_suspend_timeout ❌​F ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/resources/postgres_endpoints/remove_suspend_timeout/DMS= ❌​F ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/resources/postgres_endpoints/remove_suspend_timeout/DMS=true ❌​F ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/resources/postgres_endpoints/update_autoscaling ❌​F ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/resources/postgres_endpoints/update_autoscaling/DMS= ❌​F ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/resources/postgres_endpoints/update_autoscaling/DMS=true ❌​F ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/resources/postgres_endpoints/update_pg_settings ❌​F ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/resources/postgres_endpoints/update_pg_settings/DMS= ❌​F ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/resources/postgres_endpoints/update_pg_settings/DMS=true ❌​F ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/resources/postgres_endpoints/update_suspend_timeout ❌​F ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/resources/postgres_endpoints/update_suspend_timeout/DMS= ❌​F ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/resources/postgres_endpoints/update_suspend_timeout/DMS=true ❌​F ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/resources/postgres_projects/add_default_endpoint_settings ❌​F ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/resources/postgres_projects/add_default_endpoint_settings/DMS= ❌​F ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/resources/postgres_projects/add_default_endpoint_settings/DMS=true ❌​F ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/resources/postgres_projects/remove_history_retention ❌​F ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/resources/postgres_projects/remove_history_retention/DMS= ❌​F ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/resources/postgres_projects/remove_history_retention/DMS=true ❌​F ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/resources/postgres_projects/update_default_endpoint_suspend ❌​F ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/resources/postgres_projects/update_default_endpoint_suspend/DMS= ❌​F ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/resources/postgres_projects/update_default_endpoint_suspend/DMS=true ❌​F ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/resources/postgres_projects/update_display_name ❌​F ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/resources/postgres_projects/update_display_name/DMS= ❌​F ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/resources/postgres_projects/update_display_name/DMS=true ❌​F ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/resources/postgres_projects/update_history_retention ❌​F ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/resources/postgres_projects/update_history_retention/DMS= ❌​F ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/resources/postgres_projects/update_history_retention/DMS=true ❌​F ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/resources/postgres_roles/update ❌​F ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/resources/postgres_roles/update/DMS= ❌​F ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/resources/postgres_roles/update/DMS=true ❌​F ❌​F ❌​F ❌​F ❌​F ❌​F
🟨​ TestAccept/bundle/resources/postgres_synced_tables/recreate 🟨​K 💚​R 💚​R 💚​R 💚​R 💚​R
🟨​ TestAccept/bundle/resources/postgres_synced_tables/recreate/DMS= 🟨​K 💚​R 💚​R 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/postgres_synced_tables/recreate/DMS=true 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
❌​ TestAccept/bundle/resources/quality_monitors/change_table_name ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F
❌​ TestAccept/bundle/resources/quality_monitors/change_table_name/DMS=true ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F
❌​ TestFsCpFileToFileFileNotOverwritten ✅​p ✅​p ❌​F ✅​p ✅​p ✅​p
❌​ TestFsCpFileToFileFileNotOverwritten/local_to_uc-volumes ✅​p ✅​p ❌​F ✅​p ✅​p ✅​p
Top 50 slowest tests (at least 2 minutes):
duration env testname
15:06 aws linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=
13:55 aws-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=
12:27 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=true
11:13 aws-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=true
10:53 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=true
10:40 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=
9:49 azure-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=
9:06 gcp linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=
9:05 aws-windows-latest-4core-16gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=true
8:57 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=
8:49 aws linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=
8:49 gcp linux-2core-8gb TestAccept/bundle/resources/clusters/deploy/local_ssd_count/DMS=true
8:46 azure-windows-latest-4core-16gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=
8:42 aws linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=true
8:40 gcp linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=
8:39 azure linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=
8:38 gcp linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=
8:38 aws linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=true
8:20 azure-windows-latest-4core-16gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=true
8:19 gcp linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=true
8:08 gcp linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=true
8:04 aws-windows-latest-4core-16gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=
8:04 aws linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=true
8:01 azure-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=true
7:47 aws-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=true
7:41 gcp linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=true
7:38 gcp linux-2core-8gb TestAccept/bundle/resources/clusters/deploy/local_ssd_count/DMS=
7:36 aws linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=
7:33 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/deploy/local_ssd_count/DMS=true
7:11 aws linux-2core-8gb TestAccept/bundle/config-remote-sync/multiple_resources/DMS=true
7:10 aws-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/resize-terminated-fallback/DMS=true
7:07 azure linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=true
7:06 aws linux-2core-8gb TestAccept/bundle/resources/clusters/resize-terminated-fallback/DMS=true
6:52 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/deploy/local_ssd_count/DMS=
6:50 azure linux-2core-8gb TestAccept/bundle/invariant/no_drift/DMS=true/INPUT_CONFIG=cluster_apply_policy_default_values.yml.tmpl/READPLAN=
6:43 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=true
6:40 azure-windows-latest-4core-16gb TestAccept/bundle/invariant/delete_idempotent/DMS=/INPUT_CONFIG=cluster_apply_policy_default_values.yml.tmpl/READPLAN=
6:39 gcp linux-2core-8gb TestAccept/bundle/config-remote-sync/multiple_resources/DMS=true
6:32 aws linux-2core-8gb TestAccept/bundle/config-remote-sync/multiple_resources/DMS=
6:29 azure-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/resize-terminated-fallback/DMS=
6:22 azure linux-2core-8gb TestAccept/bundle/invariant/no_drift/DMS=true/INPUT_CONFIG=cluster_libraries.yml.tmpl/READPLAN=
6:20 aws-windows-latest-4core-16gb TestAccept/bundle/invariant/no_drift/DMS=true/INPUT_CONFIG=cluster_apply_policy_default_values.yml.tmpl/READPLAN=
6:19 azure-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/resize-terminated-fallback/DMS=true
6:14 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=
6:11 azure linux-2core-8gb TestAccept/bundle/invariant/no_drift/DMS=true/INPUT_CONFIG=cluster_libraries.yml.tmpl/READPLAN=1
6:01 azure linux-2core-8gb TestAccept/bundle/resources/clusters/deploy/update-after-create/DMS=true
5:53 gcp linux-2core-8gb TestAccept/bundle/config-remote-sync/multiple_resources/DMS=
5:48 azure-windows-latest-4core-16gb TestAccept/bundle/invariant/delete_idempotent/DMS=/INPUT_CONFIG=cluster.yml.tmpl/READPLAN=
5:47 azure-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=true
5:45 aws-windows-latest-4core-16gb TestAccept/bundle/deploy/spark-jar-task/DMS=

@denik

denik commented Nov 12, 2026

Copy link
Copy Markdown
Contributor Author

The selection cmd (acceptance/internal/selection/cmd) still prints only the changed tests, so there is no local way to see what a commit message selects without running the suite. Worth adding the named tests there, or a follow-up?

Added in #7548: go run ./acceptance/internal/selection/cmd [-cloud aws|azure|gcp] now prints the tests the last commit message names (as -run patterns) and the warnings after the changed tests. The same PR also covers your three inline comments.

— AI agent (harness_auto, opus5.5, isaac_claude)

idelafuente-oc pushed a commit to idelafuente-oc/cli that referenced this pull request Nov 14, 2026
…es, exact CloudSlow bypass (databricks#7548)

## Changes

Follow-up to [databricks#7524](databricks#7524),
addressing the review comments that came in after it merged:

- **`CloudSlow` bypass needs an exact name.** Only a name that is the
test dir itself, or one of its variants, lifts `CloudSlow`. A path above
the dir (e.g. `cloud: TestAccept/bundle`) or a bare `TestAccept` still
selects the whole subtree, but its `CloudSlow` tests stay skipped under
`-short`, which keeps the PR run inside `cloud-select`'s 2h timeout. To
support this, `selection.Named` now records per dir whether a name was
exact (`Named.Exact`).
- **Rejected lines are logged.** A line with a cloud prefix that is not
read as test names, because some word does not start with `Test` (``aws:
`TestAccept/x` ``, a trailing word, prose), now gets a `WARNING:
ignoring "<line>" ...` in the test log, next to the existing warning for
unmatched names. `ParseCommitMessage` returns these lines, and
`FromCommitMessage` now returns a `CommitSelection` with `Summary()` and
`Warnings()`, shared by the test runner and the cmd.
- **The selection cmd lists named tests.** `go run
./acceptance/internal/selection/cmd [-cloud aws|azure|gcp]` prints,
after the changed tests, the tests the last commit message names for
that run (empty `-cloud` means a local run) as `go test -run` patterns,
plus the warnings. It does this only in git mode, and only when the
message names something or has a warning, so the output for ordinary
commits is unchanged. With paths as arguments it does not read the
message.
- **Test helper reuse.** `TestLastCommitMessage` uses `runSelectionGit`,
which now returns git's output. `commit_test.go` moves to `package
selection_test` to share it.
- `acceptance/README.md` updated.

## Why

Review comments from a code owner on
[databricks#7524](databricks#7524) that auto-merge
landed before they were addressed.

## Tests

- Unit tests: rejected lines (backticks, prose) are returned trimmed,
and empty `aws:` or unknown prefixes are not reported. `Exact` is set
for a dir name and for a variant name, unset for a prefix name and for
bare `TestAccept`, and an exact name wins over a prefix name for the
same dir. `Warnings()` text.
- `acceptance/selftest/selection_exclusions` gets a case that commits a
message with names, covering the cmd's local and `-cloud gcp` output and
both warnings.
- Local, with a commit naming `aws: TestAccept/cmd/fs/cp/file-to-file`,
`cloud: TestAccept/bundle/validate/enum/DMS=true TestAccept/nope` and
``gcp: `TestAccept/cmd/fs/cp/dir-to-dir` ``:
- `DATABRICKS_TEST_SELECT_CHANGED=50 go test ./acceptance -run
'^TestAccept$' -v` ran the two named tests and logged both warnings.
- `go run ./acceptance/internal/selection/cmd` printed
`^bundle$/^validate$/^enum$/^DMS=true$` and
`^cmd$/^fs$/^cp$/^file-to-file$` plus the warnings. With `-cloud gcp` it
printed only the `cloud:` test.

This pull request and its description were written by Isaac.

---------

Co-authored-by: Isaac <no-reply@databricks.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ai-auto PRs made by AI loops & factories,

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants