Repository navigation
Distinguish resources removed from state in bundle summaries - #7496
janniklasrose wants to merge 3 commits into
Conversation
|
Commit: Regression test reportSelected 20 changed tests (limit=20, 38 not selected) 35 tests: regression=35 unreleased=0 coverage=0 failing=0
TestAccept/bundle/deploy/empty-resource-id/create/DMS=: branch ✅ | main (4bb0730) ❌ | latest (v1.26.0) ❌main (4bb0730): latest (v1.26.0): same output as main (4bb0730) TestAccept/bundle/deploy/empty-resource-id/create/DMS=true: branch ✅ | main (4bb0730) ❌ | latest (v1.26.0) ❌main (4bb0730): latest (v1.26.0): same output as main (4bb0730) TestAccept/bundle/deploy/empty-resource-id/update/DMS=: branch ✅ | main (4bb0730) ❌ | latest (v1.26.0) ❌main (4bb0730): latest (v1.26.0): same output as main (4bb0730) TestAccept/bundle/deploy/empty-resource-id/update/DMS=true: branch ✅ | main (4bb0730) ❌ | latest (v1.26.0) ❌main (4bb0730): latest (v1.26.0): TestAccept/bundle/deploy/summary-on-error/DMS=: branch ✅ | main (4bb0730) ❌ | latest (v1.26.0) ❌main (4bb0730): latest (v1.26.0): same output as main (4bb0730) TestAccept/bundle/deploy/summary-on-error/DMS=true: branch ✅ | main (4bb0730) ❌ | latest (v1.26.0) ❌main (4bb0730): latest (v1.26.0): same output as main (4bb0730) TestAccept/bundle/deployment/bind/postgres_data_api/DMS=: branch ✅ | main (4bb0730) ❌ | latest (v1.26.0) ❌main (4bb0730): latest (v1.26.0): TestAccept/bundle/destroy/summary-on-error/DMS=true: branch ✅ | main (4bb0730) ❌ | latest (v1.26.0) ❌main (4bb0730): latest (v1.26.0): same output as main (4bb0730) TestAccept/bundle/dms/failed-delete/DMS=true: branch ✅ | main (4bb0730) ❌ | latest (v1.26.0) ❌main (4bb0730): latest (v1.26.0): same output as main (4bb0730) TestAccept/bundle/dms/failed-recreate/DMS=true: branch ✅ | main (4bb0730) ❌ | latest (v1.26.0) ❌main (4bb0730): latest (v1.26.0): TestAccept/bundle/dms/failed-update-permissions/DMS=true: branch ✅ | main (4bb0730) ❌ | latest (v1.26.0) ❌main (4bb0730): latest (v1.26.0): same output as main (4bb0730) TestAccept/bundle/dms/record-failure/DMS=true: branch ✅ | main (4bb0730) ❌ | latest (v1.26.0) ❌main (4bb0730): latest (v1.26.0): same output as main (4bb0730) TestAccept/bundle/migrate/auto/apply-failure/DMS=: branch ✅ | main (4bb0730) ❌ | latest (v1.26.0) ❌main (4bb0730): latest (v1.26.0): same output as main (4bb0730) TestAccept/bundle/resource_deps/create_error/DMS=: branch ✅ | main (4bb0730) ❌ | latest (v1.26.0) ❌main (4bb0730): latest (v1.26.0): same output as main (4bb0730) TestAccept/bundle/resource_deps/create_error/DMS=true: branch ✅ | main (4bb0730) ❌ | latest (v1.26.0) ❌main (4bb0730): latest (v1.26.0): same output as main (4bb0730) TestAccept/bundle/resource_deps/missing_ingestion_definition/DMS=: branch ✅ | main (4bb0730) ❌ | latest (v1.26.0) ❌main (4bb0730): latest (v1.26.0): same output as main (4bb0730) TestAccept/bundle/resource_deps/missing_ingestion_definition/DMS=true: branch ✅ | main (4bb0730) ❌ | latest (v1.26.0) ❌main (4bb0730): latest (v1.26.0): same output as main (4bb0730) TestAccept/bundle/resource_deps/missing_map_key/DMS=: branch ✅ | main (4bb0730) ❌ | latest (v1.26.0) ❌main (4bb0730): latest (v1.26.0): same output as main (4bb0730) TestAccept/bundle/resource_deps/missing_map_key/DMS=true: branch ✅ | main (4bb0730) ❌ | latest (v1.26.0) ❌main (4bb0730): latest (v1.26.0): TestAccept/bundle/resources/apps/create_already_exists/DMS=: branch ✅ | main (4bb0730) ❌ | latest (v1.26.0) ❌main (4bb0730): latest (v1.26.0): same output as main (4bb0730) TestAccept/bundle/resources/apps/create_already_exists/DMS=true: branch ✅ | main (4bb0730) ❌ | latest (v1.26.0) ❌main (4bb0730): latest (v1.26.0): same output as main (4bb0730) TestAccept/bundle/resources/catalogs/empty-name/DMS=: branch ✅ | main (4bb0730) ❌ | latest (v1.26.0) ❌main (4bb0730): latest (v1.26.0): same output as main (4bb0730) TestAccept/bundle/resources/catalogs/empty-name/DMS=true: branch ✅ | main (4bb0730) ❌ | latest (v1.26.0) ❌main (4bb0730): latest (v1.26.0): same output as main (4bb0730) TestAccept/bundle/resources/cluster_policies/definition_and_family_conflict/DMS=: branch ✅ | main (4bb0730) ❌ | latest (v1.26.0) ❌main (4bb0730): latest (v1.26.0): 11 more failure details omitted, see the full report in the run. |
| } | ||
|
|
||
| err = d.Adapter.DoDelete(ctx, oldID, oldState) | ||
| unbound := isManagedByParent(err) |
There was a problem hiding this comment.
isManagedByParent sets unbound, so a destroy of a Lakebase branch plus its primary endpoint now prints "Destroy: 2 deleted, 1 removed from state" (postgres_endpoints/replace_existing) even though the parent delete cascade-removes the endpoint, as util.go:9 and that test's script say. Should parent-managed deletes keep counting as deleted rather than reading as left behind?
| db.RecordFailure(d.ResourceKey, oldID, err) | ||
| return err | ||
| } else { | ||
| unbound = true |
There was a problem hiding this comment.
Without deployment history this branch ignores the delete error silently, so the new "Destroy: 2 deleted, 1 removed from state" (destroy/summary-on-error, settled_failure) names no resource and gives no reason. Can it log.Warnf like the permission branch above?
❌ Integration test reportCommit: d056548
10 interesting tests: 6 flaky, 4 FAIL
Top 45 slowest tests (at least 2 minutes):
|
d056548 to
373373f
Compare
373373f to
e52d600
Compare
e52d600 to
e6b39ea
Compare
…s to CI (databricks#7501) ## Changes The main-distance check now says the review judges the code at `TARGET` and only tells the author what moved on main. Stacked PRs take `BASE` from their base branch so the parent PR is not reviewed twice. The sentence asking to regenerate generated files locally is replaced by a pointer to the CI check. ## Why Follow-up to pietern's questions on databricks#7466, plus databricks#7496 (stacked on databricks#7006) where the parent hunks showed up in the review. ## Tests Docs only. This pull request and its description were written by Isaac. --------- Co-authored-by: Isaac <no-reply@databricks.com>
Sankalp-Mittal
left a comment
There was a problem hiding this comment.
🤖 AI review from review-factory: 3 independent reviewer models (Claude Fable 5.2, Claude Opus 5.5, Claude Sonnet 5.5) with cross-verification. Only findings that reviewers agreed on are posted.
Reports deletes with ignored API errors as 'removed from state' in summaries; reviewers found parent-managed cascade deletes misreported and a minor wait double count.
Findings: 1 major, 1 minor.
|
|
||
| err = d.Adapter.DoDelete(ctx, oldID, oldState) | ||
| // A parent-managed resource may remain if its parent is kept or fails to delete. | ||
| unbound := isManagedByParent(err) |
There was a problem hiding this comment.
major: Parent-managed deletes are reported as "removed from state" although the parent's delete cascade-cleans them, with no name or reason
unbound := isManagedByParent(err) puts every MANAGED_BY_PARENT rejection in the new "removed from state" bucket. isManagedByParent's own doc (bundle/direct/util.go:9-15) says the standalone Delete can be disregarded because the parent's Delete will cascade-clean the resource. On bundle destroy the parent is always in the plan and is deleted in the same run, so these resources are in fact deleted. The new summary says the opposite: any bundle that manages a Lakebase root branch or primary endpoint now prints Destroy: N-1 deleted, 1 removed from state on a fully successful destroy. Seven goldens encode this (postgres_branches/replace_existing, postgres_endpoints/replace_existing, and five postgres_data_apis tests). Before this change the line was N deleted, which was correct for these cases.
The output also contradicts itself. In postgres_data_apis/basic the approval text says the action "will result in the deletion of the following Lakebase branches" and lists resources.postgres_branches.production, and the summary a few lines later reads Destroy: 3 deleted, 1 removed from state.
Unlike the other two unbound branches (permission denied, no history), this branch logs nothing, and destroy prints no per-resource lines (reportApplied is false at bundle/phases/destroy.go:181). The user is told one resource may have been left behind, with no resource name and no reason, and may go looking for an orphaned branch or endpoint that no longer exists. This is the open question from andrewnester on this line; the code at head is unchanged on it.
The case the PR description gives as justification (the parent is kept, or the parent's delete fails) is real, but no test covers it: the only parent-managed scenarios in the suite are destroys where the parent is deleted. rg MANAGED_BY_PARENT acceptance/bundle matches only the postgres_endpoints/replace_existing and postgres_branches/replace_existing scripts, and rg "Removed postgres" acceptance/bundle has no match, so the deploy-path line for a parent-managed resource is not covered either.
Suggested changes (either one):
- Do not set
UnboundforMANAGED_BY_PARENT. This matches howRecreatealready treats it (RecreateDeleted = HasDoDelete() && !managedByParent), which counts it as neither deleted nor unbound. - Or count a parent-managed rejection as removed from state only when its parent is not deleted in the same apply, and keep counting it as deleted otherwise (the pre-PR result).
If the current behaviour stays, at least log.Warnf the resource key as the other two unbound branches do, and add a test for the parent-kept case.
Evidence:
bundle/direct/apply.go:271-273:unbound := isManagedByParent(err)with no log call, while:279-280and:289-290pairunbound = truewithlog.Warnf.acceptance/bundle/resources/postgres_endpoints/replace_existing/out.requests.destroy.jsonrecordsDELETE .../branches/develop/endpoints/primary, thenDELETE .../branches/develop, thenDELETE .../projects/test-pg-proj-[UNIQUE_NAME].libs/testserver/postgres.go:504-514(branch delete removes its endpoints) and:273-289(project delete removes branches and endpoints) show the cascade.out.destroy.txt:16in that test now readsDestroy: 2 deleted, 1 removed from statewith noWarn:line above it.acceptance/bundle/resources/postgres_data_apis/basic/output.txt:130-132listsdelete resources.postgres_branches.productionunder the Lakebase-branch deletion warning, and:140readsDestroy: 3 deleted, 1 removed from state.
Agreed by 3 reviewer model(s).
| if err != nil { | ||
| return fmt.Errorf("deleting state id=%s: %w", oldID, err) | ||
| } | ||
| entry.Unbound = unbound |
There was a problem hiding this comment.
minor: Unbound deletes still run the delete-wait, and a wait failure counts one resource as both failed and removed from state
entry.Unbound is set before WaitAfterDelete, which still runs even though the delete was rejected and the resource is known to still exist. For resources whose wait polls until the GET fails (vector_search_index, postgres_synced_table), an ignored 403 now means polling until the timeout or cap. A non-capped wait timeout then returns an error. CountApplied and logDestroySummary count that single resource twice: Unbound++ and Failed++. The new unit case wait failure after unbinding still reports state removal ({Unbound: 1, Failed: 1}) locks in that double count.
Polling after an ignored error is not new, but this PR newly ties it to the summary counts. Returning right after entry.Unbound = unbound when unbound is true would remove both the pointless wait and the double count, since the resource was not deleted and there is nothing to wait for.
Evidence:
bundle/direct/apply.go:298setsentry.Unboundbefore thewaitCappedcall at:305-311, which returns an error at:310.bundle/direct/dresources/vector_search_index.go:225-233andpostgres_synced_table.go:157-170poll withContinues("... still exists ...")until GET errors.bundle/deployplan/plan.go:120-127hasif entry.Unbound {c.Unbound++}followed by a separateif entry.Attempted && !entry.Applied {c.Failed++};bundle/phases/destroy.go:155-162has the same pattern.bundle/deployplan/plan_test.go:121-124asserts{Unbound: 1, Failed: 1}for a single entry.
Agreed by 3 reviewer model(s).
Changes
Report ignored resource-delete errors as
N removed from statein bundle deploy/destroy summaries, with a matching per-resource deploy message. Warn with the resource and error when deletion is ignored without deployment history. This builds on #7006.Why
The engine can intentionally drop a resource from bundle state after a permission error or another ignored delete error. Reporting it as deleted incorrectly suggests the backend resource is gone. Parent-managed delete rejections also count as state removal: the parent can be retained or fail to delete, even though a successful parent deletion may later cascade-remove the resource.
Tests
Acceptance tests cover permission-denied deletes during deploy and destroy, and ignored deletes without deployment history. Follow-up API reads confirm the resources still exist. Unit coverage includes state removal followed by a wait failure.
Local formatting, repository checks, full lint, generators, and changed acceptance tests under Go 1.27.3 pass. Reviewed locally with
.agents/skills/pr-reviewand addressed the findings. The final local full suite passed all unit tests and all acceptance cases after retries except the unchanged completion test, which exits with SIGPIPE. CI results are pending for the latest review corrections.This PR was written with Codex.