Skip to content

Fail the invariant coverage test when a config template is not in the matrix - #7353

Draft
janniklasrose wants to merge 1 commit into
mainfrom
janniklasrose/invariant-config-registration
Draft

janniklasrose wants to merge 1 commit into
mainfrom
janniklasrose/invariant-config-registration

Conversation

@janniklasrose

Copy link
Copy Markdown
Member

Changes

Adds TestInvariantConfigsListed next to TestInvariantConfigsCoverage in acceptance/invariant_test.go. It reads acceptance/bundle/invariant/test.toml through the acceptance config loader and asserts that every *.yml.tmpl under acceptance/bundle/invariant/configs/ is listed in EnvMatrix.INPUT_CONFIG, unless it is named in an explicit allowlist, and that every matrix entry names an existing template.

The allowlist records the templates that exist on main today without a matrix entry. It may only shrink: to run one of them, add it to the matrix and remove it from the list. No matrix entries or goldens change in this PR.

Why

A template that is not in the matrix never runs, and the existing coverage test only checks that a file exists per resource type. Several templates were in that state without anything flagging it.

Tests

  • go test ./acceptance -run 'TestInvariant' -count=1: both tests pass.
  • Removing one allowlist entry makes the new test fail with a message naming the template and both files, then passes again once restored.

This pull request and its description were written by Isaac.

… matrix

A config template under bundle/invariant/configs that is missing from INPUT_CONFIG never runs, and nothing flagged it. The new test requires every template to be listed or explicitly allowlisted, and every INPUT_CONFIG entry to name an existing file.

Co-authored-by: Isaac <no-reply@databricks.com>
@janniklasrose janniklasrose added the new-resource-work Skills, docs and small refactors for onboarding new bundle resources label Nov 4, 2026
@eng-dev-ecosystem-bot

Copy link
Copy Markdown
Collaborator

Integration test report

Commit: c349605

Run: 41860763794

Env ✅​pass 🙈​skip Time
✅​ aws linux-2core-8gb 272 15 7:02
✅​ aws-windows-latest-4core-16gb 274 13 7:08
✅​ azure linux-2core-8gb 271 15 7:04
✅​ azure-windows-latest-4core-16gb 273 13 7:01
✅​ gcp linux-2core-8gb 272 15 6:53
✅​ gcp-windows-latest-4core-16gb 274 13 7:25
Top 3 slowest tests (at least 2 minutes):
duration env testname
3:31 gcp-windows-latest-4core-16gb TestAccept
3:25 aws-windows-latest-4core-16gb TestAccept
3:19 azure-windows-latest-4core-16gb TestAccept

@denik denik added the test-only-fix PRs that update failing of flaky tests without changing CLI code. label Nov 11, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

new-resource-work Skills, docs and small refactors for onboarding new bundle resources test-only-fix PRs that update failing of flaky tests without changing CLI code.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants