Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 21 additions & 0 deletions .github/labeler-test-only.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
# Configuration for the test-only-fix step of the "PR Labeler" workflow
# (.github/workflows/labeler.yml). Same format as .github/labeler.yml. It is a
# separate file because this step runs with sync-labels, so the label is
# removed again once a later push adds files outside this list.
#
# any-glob-to-all-files needs a single glob that matches every changed file,
# so all test-only paths are alternatives of one brace expansion.
#
# The bundle/, internal/ and libs/ entries are packages the CLI binary does not
# import. TestLabelerTestOnlyDirsNotInBinary in internal/build checks this.

# The list-files API returns at most 3000 files, sorted by path. acceptance/
# sorts before bundle/, cmd/ and libs/, so a truncated list could hide CLI
# changes. Larger PRs skip this step and keep their current labels.
# https://docs.github.com/en/rest/pulls/pulls#list-pull-requests-files
max-files-changed: 2999

test-only-fix:
- changed-files:
- any-glob-to-all-files:
- "{**/*_test.go,**/testdata/**,acceptance/**,integration/**,bundle/tests/**,bundle/internal/bundletest/**,internal/mocks/**,internal/testarchive/**,internal/testcli/**,internal/testutil/**,libs/appproxy/testutil/**,libs/cmdio/cmdiotest/**,libs/dyn/dynassert/**,libs/jsonschema/test_types/**,libs/testdiff/**,libs/testfile/**,libs/testproxy/**,libs/testserver/**,python/databricks_tests/**,python/codegen/codegen_tests/**}"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The package entries are a snapshot of go list -deps .; nothing flags it when one of them later gets imported by the binary, and from then on a PR changing it is labeled test-only. Is it worth a test next to internal/build/license_test.go that reads this glob and asserts none of its directory entries appear in the binary's deps, or is a stale entry cheap enough to catch by hand?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added TestLabelerTestOnlyDirsNotInBinary next to license_test.go (85f9514). It runs go list -deps for all 6 release targets and fails if any directory entry is imported.

10 changes: 9 additions & 1 deletion .github/workflows/labeler.yml
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
name: PR Labeler

# Applies labels to PRs based on the files they touch, using the glob->label
# mapping in .github/labeler.yml.
# mappings in .github/labeler.yml and .github/labeler-test-only.yml.
#
# WARNING:
# This workflow uses pull_request_target so it can label PRs from forks (the
Expand Down Expand Up @@ -32,3 +32,11 @@ jobs:
repo-token: ${{ secrets.GITHUB_TOKEN }}
# Only add labels; never remove labels that were applied manually or by other automation.
sync-labels: false

- uses: actions/labeler@bf12e9b00b37c5c0ca2b87b79b2daf7891dbda13 # v7.0.0
with:
repo-token: ${{ secrets.GITHUB_TOKEN }}
configuration-path: .github/labeler-test-only.yml
# test-only-fix claims the PR changes no CLI code, so this step owns it: a push that
# changes any file outside the config removes it, even when it was added by hand.
sync-labels: true

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sync-labels also strips a test-only-fix that someone added by hand to a PR outside the glob, on its next push: a CI-only acceptance fix like #7181 (.github/workflows/regression-report.yml) or #7192 (libs/retry/retry.go, added to silence acceptance output) would lose it. Is that intended, given the label description covers fixes for failing or flaky tests generally? If so, could the comment on line 40 say the label is labeler-managed and a manual add does not survive a push?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Intended: for a label that claims no CLI code changed, staying on after CLI code is added is the worse failure. The comment now says this step owns the label and a hand-added one does not survive a push (85f9514). #7192 touches libs/retry/retry.go, which is CLI code, so it falls outside the label description. For CI-only fixes like #7181 we can add the test-reporting workflows to the glob if you think it is worth it.

1 change: 1 addition & 0 deletions Taskfile.yml
Original file line number Diff line number Diff line change
Expand Up @@ -447,6 +447,7 @@ tasks:
# internal/build reads these at test time.
- NOTICE
- .codegen/_openapi_sha
- .github/labeler-test-only.yml
# bundle/internal/schema TestRequiredAnnotationsForNewFields reads these.
- bundle/internal/schema/annotations*.yml
generates:
Expand Down
67 changes: 67 additions & 0 deletions internal/build/labeler_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
package build

import (
"os"
"os/exec"
"strings"
"testing"

"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"go.yaml.in/yaml/v3"
)

// TestLabelerTestOnlyDirsNotInBinary fails when the CLI binary imports a package
// under a directory that .github/labeler-test-only.yml treats as test-only, since
// a PR changing that directory would then be labeled test-only-fix.
func TestLabelerTestOnlyDirsNotInBinary(t *testing.T) {
raw, err := os.ReadFile("../../.github/labeler-test-only.yml")
require.NoError(t, err)

var config struct {
TestOnlyFix []struct {
ChangedFiles []struct {
AnyGlobToAllFiles []string `yaml:"any-glob-to-all-files"`
} `yaml:"changed-files"`
} `yaml:"test-only-fix"`
}
require.NoError(t, yaml.Unmarshal(raw, &config))
glob := config.TestOnlyFix[0].ChangedFiles[0].AnyGlobToAllFiles[0]

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This reads only the first entry at each level, so a second glob appended to any-glob-to-all-files (or a second changed-files block) would widen what the labeler treats as test-only without this test ever seeing it. Can we require.Len(..., 1) at each level, or loop over every entry?


// Directory entries look like "libs/testserver/**". Entries such as
// "**/*_test.go" match by file name and are skipped.
var dirs []string
for alt := range strings.SplitSeq(strings.Trim(glob, "{}"), ",") {
dir, ok := strings.CutSuffix(alt, "/**")
if ok && !strings.Contains(dir, "*") {
dirs = append(dirs, dir)
}
}
require.NotEmpty(t, dirs)

// Release targets from .goreleaser.yaml; some files build only on some of them.
for _, target := range []string{"linux/amd64", "linux/arm64", "darwin/amd64", "darwin/arm64", "windows/amd64", "windows/arm64"} {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This duplicates the goos × goarch matrix from .goreleaser.yaml, so adding a release target there leaves this guard one target short. Since the test already parses YAML, is reading builds[0].goos/goarch from .goreleaser.yaml worth it, or is the list stable enough to keep inline?

t.Run(target, func(t *testing.T) {
t.Parallel()
goos, goarch, _ := strings.Cut(target, "/")
// go test does not see the sources go list reads, so a cached local pass can be
// stale. CI reruns it because a fresh checkout changes the config file's mtime.
cmd := exec.CommandContext(t.Context(), "go", "list", "-deps", "-f", "{{.ImportPath}}", ".")
cmd.Dir = "../.."
cmd.Env = append(os.Environ(), "GOOS="+goos, "GOARCH="+goarch, "CGO_ENABLED=0")
out, err := cmd.Output()
require.NoError(t, err)

for pkg := range strings.FieldsSeq(string(out)) {
rel, ok := strings.CutPrefix(pkg, "github.com/databricks/cli/")
if !ok {
continue
}
for _, dir := range dirs {
assert.False(t, rel == dir || strings.HasPrefix(rel, dir+"/"),
"CLI binary imports %s, but .github/labeler-test-only.yml lists %s/** as test-only", pkg, dir)
}
}
})
}
}
Loading