Repository navigation
acceptance: invariant that every deployed resource generates back into a valid bundle - #7440
janniklasrose wants to merge 3 commits into
Conversation
…o a valid bundle New invariant target bundle/invariant/generate runs over the same per-resource config matrix as no_drift: deploy, then `bundle generate <group> --existing-id` for every resource in state into a fresh bundle, then `bundle validate` it. Local only; the round-trip under test is the CLI's own. generate_from_state.py drives the loop from resources.json and is silent on success so all configs record the same output. Types the round-trip check rejects today (secret scopes, secrets, vector search indexes) and the Postgres endpoint, whose GET returns no spec, are excluded with TODOs; app sources are not served by the fake workspace. Co-authored-by: Isaac <no-reply@databricks.com>
|
Commit: Regression test reportSelected 7 changed tests (limit=20, 0 not selected) 69 tests: regression=68 unreleased=0 coverage=1 failing=0
TestAccept/bundle/generate/catalog/DMS=: branch ✅ | main (6a8d9bf) ❌ | latest (v1.25.0) ❌main (6a8d9bf): latest (v1.25.0): same output as main (6a8d9bf) TestAccept/bundle/generate/catalog/DMS=true: branch ✅ | main (6a8d9bf) ❌ | latest (v1.25.0) ❌main (6a8d9bf): latest (v1.25.0): same output as main (6a8d9bf) TestAccept/bundle/help/bundle-generate-dashboard/DMS=: branch ✅ | main (6a8d9bf) ❌ | latest (v1.25.0) ❌main (6a8d9bf): latest (v1.25.0): same output as main (6a8d9bf) TestAccept/bundle/help/bundle-generate-dashboard/DMS=true: branch ✅ | main (6a8d9bf) ❌ | latest (v1.25.0) ❌main (6a8d9bf): latest (v1.25.0): same output as main (6a8d9bf) TestAccept/bundle/help/bundle-generate-job/DMS=: branch ✅ | main (6a8d9bf) ❌ | latest (v1.25.0) ❌main (6a8d9bf): latest (v1.25.0): same output as main (6a8d9bf) TestAccept/bundle/help/bundle-generate-job/DMS=true: branch ✅ | main (6a8d9bf) ❌ | latest (v1.25.0) ❌main (6a8d9bf): latest (v1.25.0): same output as main (6a8d9bf) TestAccept/bundle/help/bundle-generate-pipeline/DMS=: branch ✅ | main (6a8d9bf) ❌ | latest (v1.25.0) ❌main (6a8d9bf): latest (v1.25.0): same output as main (6a8d9bf) TestAccept/bundle/help/bundle-generate-pipeline/DMS=true: branch ✅ | main (6a8d9bf) ❌ | latest (v1.25.0) ❌main (6a8d9bf): latest (v1.25.0): same output as main (6a8d9bf) TestAccept/bundle/help/bundle-generate/DMS=: branch ✅ | main (6a8d9bf) ❌ | latest (v1.25.0) ❌main (6a8d9bf): latest (v1.25.0): same output as main (6a8d9bf) TestAccept/bundle/help/bundle-generate/DMS=true: branch ✅ | main (6a8d9bf) ❌ | latest (v1.25.0) ❌main (6a8d9bf): latest (v1.25.0): same output as main (6a8d9bf) TestAccept/bundle/invariant/generate/INPUT_CONFIG=alert.yml.tmpl: branch ✅ | main (6a8d9bf) ❌ | latest (v1.25.0) ❌main (6a8d9bf): latest (v1.25.0): TestAccept/bundle/invariant/generate/INPUT_CONFIG=catalog.yml.tmpl: branch ✅ | main (6a8d9bf) ❌ | latest (v1.25.0) ❌main (6a8d9bf): latest (v1.25.0): TestAccept/bundle/invariant/generate/INPUT_CONFIG=catalog_optional_fields.yml.tmpl: branch ✅ | main (6a8d9bf) ❌ | latest (v1.25.0) ❌main (6a8d9bf): latest (v1.25.0): TestAccept/bundle/invariant/generate/INPUT_CONFIG=cluster.yml.tmpl: branch ✅ | main (6a8d9bf) ❌ | latest (v1.25.0) ❌main (6a8d9bf): latest (v1.25.0): TestAccept/bundle/invariant/generate/INPUT_CONFIG=cluster_apply_policy_default_values.yml.tmpl: branch ✅ | main (6a8d9bf) ❌ | latest (v1.25.0) ❌main (6a8d9bf): latest (v1.25.0): 53 more failure details omitted, see the full report in the run. |
✅ Integration test reportCommit: 2d33f67
Top 4 slowest tests (at least 2 minutes):
|
Co-authored-by: Isaac <no-reply@databricks.com>
…rose/bundle-generate-invariant
Changes
Adds the invariant target
acceptance/bundle/invariant/generate. It runs over the same per-resource config matrix asno_drift: deploy the minimal config, then for every resource recorded in state runbundle generate <group> --existing-id <id> --key <key>into a fresh bundle, thenbundle validatethat bundle. The target is local only (Cloud = false): the round-trip from deployment state to configuration is the CLI's own, and the fake server exercises it as well as a workspace would.acceptance/bin/generate_from_state.pydrives the loop fromresources.json. It is silent on success so every config records the same two lines, and on failure it prints the failing command and its output, in the style of the other invariants.The job generator downloads the notebooks a job references, so the script imports
/Shared/notebookinto the fake workspace first; the invariant job configs all point there.Excluded configs
job_run*: job runs have no generate command.app: the app generator downloads the app's source files, which the fake workspace does not serve.secret_scope*(state usesscope/scope_backend_type),secret(PrepareState clears metadata RemapState keeps),vector_search_index(same forendpoint_uuid).postgres_endpoint: GET does not return the spec, so the generatedendpoint_typeis empty and validate warns.Each exclusion is a
TODOintest.toml; dropping it is the acceptance criterion for the adapter fix.Why
The base PR makes
bundle generatecover every resource type by deriving it from the direct-engine adapters. This target is what proves the derivation per type without per-type test code, and it found the four TODOs above on its first run.Tests
57 configs pass locally.
./task wsand./task lint-pythonpass.This pull request and its description were written by Isaac.