Repository navigation
testmask: run tests named in the last commit message - #7524
Conversation
Lines like "aws: TestAccept/<path>" or "cloud: TestAccept/<path>" in the last commit message of a PR add those tests to the PR cloud run on the named clouds and to the local selection. See acceptance/README.md. aws: TestAccept/cmd/fs/cp/file-to-file Co-authored-by: Isaac <no-reply@databricks.com>
…sk trigger aws: TestAccept/cmd/fs/cp/file-to-file Co-authored-by: Isaac <no-reply@databricks.com>
✅ Integration test reportCommit: ad0826a
Top 6 slowest tests (at least 2 minutes):
|
andrewnester
left a comment
There was a problem hiding this comment.
The selection cmd (acceptance/internal/selection/cmd) still prints only the changed tests, so there is no local way to see what a commit message selects without running the suite. Worth adding the named tests there, or a follow-up?
|
|
||
| // CloudSlow only narrows an already-enabled cloud run: skip it under -short. | ||
| if isTruePtr(config.CloudSlow) && testing.Short() { | ||
| if isTruePtr(config.CloudSlow) && testing.Short() && !named { |
There was a problem hiding this comment.
A prefix name like cloud: TestAccept lifts CloudSlow for every dir under it, more than the -short nightly runs and inside cloud-select's 2h timeout. Should the bypass need an exact dir or variant name?
There was a problem hiding this comment.
Agreed. Fixed in #7548: only a name that is the dir itself or one of its variants lifts CloudSlow (new Named.Exact). A prefix such as cloud: TestAccept or TestAccept/bundle still selects the subtree, but its CloudSlow tests stay skipped under -short.
— AI agent (harness_auto, opus5.5, isaac_claude)
| continue | ||
| } | ||
| words := strings.Fields(m[2]) | ||
| if len(words) == 0 || slices.ContainsFunc(words, func(word string) bool { return !strings.HasPrefix(word, "Test") }) { |
There was a problem hiding this comment.
A line like aws: `TestAccept/x` (backticks, or a trailing word) is dropped as prose with no warning, unlike an unmatched name. Can those lines be logged too, so a silent no-op is visible?
There was a problem hiding this comment.
Agreed. Fixed in #7548: a cloud-prefixed line that is not read as names (backticks, a trailing word, prose) now logs WARNING: ignoring "<line>" ... next to the unmatched-name warning, and the selection cmd prints it too.
— AI agent (harness_auto, opus5.5, isaac_claude)
| } | ||
| t.Chdir(t.TempDir()) | ||
|
|
||
| git := func(args ...string) string { |
There was a problem hiding this comment.
runSelectionGit in selection_test.go already runs git with a fixed identity for this package's tests. Can this reuse it instead of a second git closure plus config calls?
There was a problem hiding this comment.
Agreed. Done in #7548: runSelectionGit now returns git's output and TestLastCommitMessage uses it; commit_test.go moved to package selection_test to share it.
— AI agent (harness_auto, opus5.5, isaac_claude)
❌ Integration test reportCommit: 69de469
55 interesting tests: 43 FAIL, 9 RECOVERED, 3 KNOWN
Top 50 slowest tests (at least 2 minutes):
|
Added in #7548: — AI agent (harness_auto, opus5.5, isaac_claude) |
…es, exact CloudSlow bypass (databricks#7548) ## Changes Follow-up to [databricks#7524](databricks#7524), addressing the review comments that came in after it merged: - **`CloudSlow` bypass needs an exact name.** Only a name that is the test dir itself, or one of its variants, lifts `CloudSlow`. A path above the dir (e.g. `cloud: TestAccept/bundle`) or a bare `TestAccept` still selects the whole subtree, but its `CloudSlow` tests stay skipped under `-short`, which keeps the PR run inside `cloud-select`'s 2h timeout. To support this, `selection.Named` now records per dir whether a name was exact (`Named.Exact`). - **Rejected lines are logged.** A line with a cloud prefix that is not read as test names, because some word does not start with `Test` (``aws: `TestAccept/x` ``, a trailing word, prose), now gets a `WARNING: ignoring "<line>" ...` in the test log, next to the existing warning for unmatched names. `ParseCommitMessage` returns these lines, and `FromCommitMessage` now returns a `CommitSelection` with `Summary()` and `Warnings()`, shared by the test runner and the cmd. - **The selection cmd lists named tests.** `go run ./acceptance/internal/selection/cmd [-cloud aws|azure|gcp]` prints, after the changed tests, the tests the last commit message names for that run (empty `-cloud` means a local run) as `go test -run` patterns, plus the warnings. It does this only in git mode, and only when the message names something or has a warning, so the output for ordinary commits is unchanged. With paths as arguments it does not read the message. - **Test helper reuse.** `TestLastCommitMessage` uses `runSelectionGit`, which now returns git's output. `commit_test.go` moves to `package selection_test` to share it. - `acceptance/README.md` updated. ## Why Review comments from a code owner on [databricks#7524](databricks#7524) that auto-merge landed before they were addressed. ## Tests - Unit tests: rejected lines (backticks, prose) are returned trimmed, and empty `aws:` or unknown prefixes are not reported. `Exact` is set for a dir name and for a variant name, unset for a prefix name and for bare `TestAccept`, and an exact name wins over a prefix name for the same dir. `Warnings()` text. - `acceptance/selftest/selection_exclusions` gets a case that commits a message with names, covering the cmd's local and `-cloud gcp` output and both warnings. - Local, with a commit naming `aws: TestAccept/cmd/fs/cp/file-to-file`, `cloud: TestAccept/bundle/validate/enum/DMS=true TestAccept/nope` and ``gcp: `TestAccept/cmd/fs/cp/dir-to-dir` ``: - `DATABRICKS_TEST_SELECT_CHANGED=50 go test ./acceptance -run '^TestAccept$' -v` ran the two named tests and logged both warnings. - `go run ./acceptance/internal/selection/cmd` printed `^bundle$/^validate$/^enum$/^DMS=true$` and `^cmd$/^fs$/^cp$/^file-to-file$` plus the warnings. With `-cloud gcp` it printed only the `cloud:` test. This pull request and its description were written by Isaac. --------- Co-authored-by: Isaac <no-reply@databricks.com>
Changes
PR runs only cover the tests a branch changed. With this PR, a test can also be requested by naming it in the last commit message of the PR:
aws:,azure:,gcp:select the test on that cloud only,cloud:on every cloud. The prefix is lowercase and must start the line (surrounding whitespace is ignored). A line can list several whitespace-separated names. Lines with other prefixes are ignored, and so is any line with a word that does not start withTest, so prose such asaws: fix flaky TestAccept retriescannot select the whole suite.TestAccept/<path>selects every test dir equal to or under<path>/(path-segment prefix, sojobsdoes not matchjobsx), with all variants.TestAccept/<dir>/KEY=value[/KEY=value...]selects only the matching variants, asgo testnames them. EachKEY=valuemust appear in one of the dir's variants (perout.test.toml), so a misspelled key or value is reported instead of running every variant or none. A bareTestAcceptselects everything. Names that match no acceptance test get a warning in the test log and are otherwise ignored.DATABRICKS_TEST_SELECT_CHANGEDselection (PR cloud run,./task local-select) and the windows/macOS PR subset (TESTS_SELECT_SUBSET_PCT), and they don't count toward those limits. Full runs (Linux PR cell, nightly) are unchanged.Cloud = trueand a matchingCloudEnvs. It does run even if it isCloudSlow.tools/testmask(only forpull_requestevents, viaGITHUB_EVENT_NAME) adds thetesttarget when the head commit has such a line, so the test jobs and the cloud trigger run even for a PR whose files would not trigger them. The merge queue ignores the message. Failing to read the message only logs a warning.Where it lives:
acceptance/internal/selection/commit.go: parser, name resolution and reading the last commit message. On theMerge <head> into <base>commit GitHub creates forrefs/pull/N/merge(which both the PR test job and the cloud run check out), it reads the PR head commit's message instead.acceptance/acceptance_test.go,acceptance/subset_test.go: wiring.tools/testmask: trigger detection (a small regex that mirrors the parser; the tools module cannot importacceptance/internal).acceptance/README.md.Why
You can then run specific tests on specific clouds from a PR (for example, to check a fix for a cloud-only failure) by pushing an empty commit, without editing test files or running the full cloud suite.
Decisions:
cli-isolated-pr.ymlruns this repo'scloud-selecttask (aliasintegration-short-skiplocal) from the PR checkout withCLOUD_ENVset, so the selection code at the PR's commit already knows which cloud it is on (getCloudEnvBase:aws/ucws-> aws,gcp/gcp-ucws-> gcp,azure-> azure).TestAccept/bundle/resources/jobs) is a directory of many tests.git merge maincommit at HEAD hides the names locally; CI is unaffected because it reads the PR head commit. testmask repeats the prefix list because the tools module cannot importacceptance/internal.CloudSlowbecause someone asked for them by name. They do not bypassCloud = false: such tests cannot run on cloud.Tests
DATABRICKS_TEST_SELECT_CHANGED=50 go test ./acceptance -run '^TestAccept$'with a commit namingaws: TestAccept/cmd/fs/cp/file-to-file,cloud: TestAccept/bundle/validate/enum/DMS=trueandgcp: TestAccept/cmd/fs/cp/dir-to-dirran all three, and forenumonly theDMS=truevariant.testme-aws -short, same commit): selectedfile-to-file(passed) andenum(skipped,Cloud = false). The gcp-only name was not selected.aws: TestAccept/cmd/fs/cp/file-to-file, so this PR's own cloud run exercises the feature.This pull request and its description were written by Isaac.