Skip to content

Label PRs that only change tests with test-only-fix - #7462

Merged
denik merged 2 commits into
mainfrom
denik/test-only-fix
Nov 11, 2026
Merged

denik merged 2 commits into
mainfrom
denik/test-only-fix

Conversation

@denik

@denik denik commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Changes

Adds a second step to the PR labeler, configured in .github/labeler-test-only.yml. It applies test-only-fix when every changed file is a test: _test.go, testdata/, acceptance/, integration/, Python tests, or a test-only package the CLI binary doesn't import (libs/testserver, internal/testutil, ...).

This label is synced: a later push that adds non-test files removes it. Other labels stay add-only. PRs with 3000+ files are skipped because the list-files API truncates there and acceptance/ sorts first.

Why

So test-only fixes can be told apart from CLI changes without manual labeling.

Tests

The test-only package list comes from packages missing from go list -deps . on darwin, linux and windows. I checked the config with the js-yaml and minimatch versions labeler v7 uses:

  • No matched file is non-test code in a binary package or a file embedded in the binary.
  • Of the last 400 commits on main, 58 would have been labeled, all test-only.

TestLabelerTestOnlyDirsNotInBinary (internal/build) fails if the binary later imports a listed package, on any release target.

The labeler reads its config from main, so this only takes effect after merge.

This pull request and its description were written by Isaac.

The PR labeler gets a second step with its own config. A PR gets the label
when every changed file is a test: _test.go, testdata, acceptance/,
integration/, Python tests, or a package the CLI binary does not import.
This step syncs, so a later push that adds non-test files removes the label.

Co-authored-by: Isaac <no-reply@databricks.com>
@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Collaborator

✅ Integration test report

Commit: 85f9514

Run: 43027644597

Env ✅​pass 🙈​skip Time
✅​ aws linux-2core-8gb 271 15 5:13
✅​ aws-windows-latest-4core-16gb 273 13 3:00
✅​ azure linux-2core-8gb 270 15 5:11
✅​ azure-windows-latest-4core-16gb 272 13 3:53
✅​ gcp linux-2core-8gb 271 15 5:11
✅​ gcp-windows-latest-4core-16gb 273 13 3:56
Top 6 slowest tests (at least 2 minutes):
duration env testname
3:57 aws linux-2core-8gb TestAccept
3:52 azure linux-2core-8gb TestAccept
3:49 gcp-windows-latest-4core-16gb TestAccept
3:47 azure-windows-latest-4core-16gb TestAccept
3:46 gcp linux-2core-8gb TestAccept
2:58 aws-windows-latest-4core-16gb TestAccept

@denik
denik marked this pull request as ready for review November 11, 2026 12:45
@denik
denik requested a review from a team as a code owner November 11, 2026 12:45
@denik
denik requested a review from andrewnester November 11, 2026 12:45

@andrewnester andrewnester left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: the workflow header (.github/workflows/labeler.yml lines 3-4) still says the glob->label mapping lives in .github/labeler.yml only; worth mentioning the second config there?

repo-token: ${{ secrets.GITHUB_TOKEN }}
configuration-path: .github/labeler-test-only.yml
# test-only-fix claims the PR changes no CLI code, so remove it once that stops being true.
sync-labels: true

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sync-labels also strips a test-only-fix that someone added by hand to a PR outside the glob, on its next push: a CI-only acceptance fix like #7181 (.github/workflows/regression-report.yml) or #7192 (libs/retry/retry.go, added to silence acceptance output) would lose it. Is that intended, given the label description covers fixes for failing or flaky tests generally? If so, could the comment on line 40 say the label is labeler-managed and a manual add does not survive a push?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Intended: for a label that claims no CLI code changed, staying on after CLI code is added is the worse failure. The comment now says this step owns the label and a hand-added one does not survive a push (85f9514). #7192 touches libs/retry/retry.go, which is CLI code, so it falls outside the label description. For CI-only fixes like #7181 we can add the test-reporting workflows to the glob if you think it is worth it.

test-only-fix:
- changed-files:
- any-glob-to-all-files:
- "{**/*_test.go,**/testdata/**,acceptance/**,integration/**,bundle/tests/**,bundle/internal/bundletest/**,internal/mocks/**,internal/testarchive/**,internal/testcli/**,internal/testutil/**,libs/appproxy/testutil/**,libs/cmdio/cmdiotest/**,libs/dyn/dynassert/**,libs/jsonschema/test_types/**,libs/testdiff/**,libs/testfile/**,libs/testproxy/**,libs/testserver/**,python/databricks_tests/**,python/codegen/codegen_tests/**}"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The package entries are a snapshot of go list -deps .; nothing flags it when one of them later gets imported by the binary, and from then on a PR changing it is labeled test-only. Is it worth a test next to internal/build/license_test.go that reads this glob and asserts none of its directory entries appear in the binary's deps, or is a stale entry cheap enough to catch by hand?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added TestLabelerTestOnlyDirsNotInBinary next to license_test.go (85f9514). It runs go list -deps for all 6 release targets and fails if any directory entry is imported.

Comment thread .github/labeler-test-only.yml Outdated
# any-glob-to-all-files needs a single glob that matches every changed file,
# so all test-only paths are alternatives of one brace expansion.
#
# The libs/ and internal/ entries are packages the CLI binary does not import

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This list also has bundle/tests/** and bundle/internal/bundletest/**, which are neither libs/ nor internal/ entries. Say "The bundle/, internal/ and libs/ entries"?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 85f9514.

TestLabelerTestOnlyDirsNotInBinary fails when the binary imports a package
under a directory entry of the test-only-fix glob, for every release target.
Also clarify comments: name the second config in the workflow header, say a
manually added label does not survive a push, and include bundle/ entries.

Co-authored-by: Isaac <no-reply@databricks.com>
@denik
denik requested a review from a team as a code owner November 11, 2026 13:15
@denik
denik requested a review from maksim-zuev-db November 11, 2026 13:15
@denik

denik commented Nov 11, 2026

Copy link
Copy Markdown
Contributor Author

Re the workflow header nit: done in 85f9514, the header now names both configs.

@denik
denik added this pull request to the merge queue Nov 11, 2026
Merged via the queue into main with commit e6045a1 Nov 11, 2026
38 checks passed
@denik
denik deleted the denik/test-only-fix branch November 11, 2026 13:49
} `yaml:"test-only-fix"`
}
require.NoError(t, yaml.Unmarshal(raw, &config))
glob := config.TestOnlyFix[0].ChangedFiles[0].AnyGlobToAllFiles[0]

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This reads only the first entry at each level, so a second glob appended to any-glob-to-all-files (or a second changed-files block) would widen what the labeler treats as test-only without this test ever seeing it. Can we require.Len(..., 1) at each level, or loop over every entry?

require.NotEmpty(t, dirs)

// Release targets from .goreleaser.yaml; some files build only on some of them.
for _, target := range []string{"linux/amd64", "linux/arm64", "darwin/amd64", "darwin/arm64", "windows/amd64", "windows/arm64"} {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This duplicates the goos × goarch matrix from .goreleaser.yaml, so adding a release target there leaves this guard one target short. Since the test already parses YAML, is reading builds[0].goos/goarch from .goreleaser.yaml worth it, or is the list stable enough to keep inline?

tedjt pushed a commit to tedjt/cli that referenced this pull request Nov 12, 2026
…abricks#7464)

## Changes
Pass 8 of the pr-review skill listed new doc comments with a Go-only
grep. This widens the pass to Python, shell and YAML (the languages
`.agents/rules/comments.md` covers), counts `acceptance/bin/*.py`
docstrings and `--help` text and the doc comment of a new test function
in test-only changes, and asks the reviewer to check the other callers
of an API when a new comment claims a quirk of it.

## Why
In reviews of databricks#7187, databricks#7462, databricks#6923, databricks#3518, databricks#6714 and databricks#7184 the pass
produced an empty table on exactly the changes where the comment was the
main thing to judge: a YAML labeler config, a Python-only comment block,
a test-only invariant test, a shared acceptance helper. The findings on
those PRs came from reading `comments.md` by hand instead. On databricks#7184 a
new comment asserted an API rejection that another caller of the same
endpoint contradicts.

## Tests
Documentation only, no code changes.

This pull request and its description were written by Isaac.

Co-authored-by: Isaac <no-reply@databricks.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants