Repository navigation
direct: retry transient errors when creating postgres resources - #7557
Merged
Merged
Conversation
Creating a postgres project sometimes fails with 500 INTERNAL_ERROR, either on the create request or on polling the create operation. DoCreate failures are not retried, unlike update and delete. - Poll the create operation again on transient errors for all postgres resources: the create request was accepted, so polling cannot duplicate it. - Mark a failed project create request as retry-safe: it is keyed by project_id, so a repeat cannot create a second project. cloud: TestAccept/bundle/resources/postgres_projects/basic TestAccept/bundle/resources/postgres_endpoints/basic Co-authored-by: Isaac <no-reply@databricks.com>
cloud: TestAccept/bundle/resources/postgres_projects/basic TestAccept/bundle/resources/postgres_endpoints/basic Co-authored-by: Isaac <no-reply@databricks.com>
cloud: TestAccept/bundle/resources/postgres_projects/basic TestAccept/bundle/resources/postgres_endpoints/basic Co-authored-by: Isaac <no-reply@databricks.com>
Contributor
|
Commit: Regression test reportSelected 1 changed tests (limit=20, 0 not selected) 6 tests: regression=2 unreleased=0 coverage=4 failing=0
TestAccept/bundle/resources/postgres_projects/create_transient_500/DMS=: branch ✅ | main (b69a249) ❌ | latest (v1.26.0) ❌main (b69a249): latest (v1.26.0): same output as main (b69a249) TestAccept/bundle/resources/postgres_projects/create_transient_500/DMS=true: branch ✅ | main (b69a249) ❌ | latest (v1.26.0) ❌main (b69a249): latest (v1.26.0): |
cloud: TestAccept/bundle/resources/postgres_projects/basic TestAccept/bundle/resources/postgres_endpoints/basic Co-authored-by: Isaac <no-reply@databricks.com>
Collaborator
✅ Integration test reportCommit: 37f3075
Top 18 slowest tests (at least 2 minutes):
|
denik
marked this pull request as ready for review
November 12, 2026 14:36
andrewnester
self-requested a review
November 12, 2026 18:14
andrewnester
left a comment
Contributor
There was a problem hiding this comment.
No blocking findings from a full pass; I'll take a manual look at the direct changes before approving.
andrewnester
approved these changes
Nov 13, 2026
Collaborator
❌ Integration test reportCommit: cf85b49
235 interesting tests: 154 MISS, 69 FAIL, 9 RECOVERED, 3 KNOWN
Top 50 slowest tests (at least 2 minutes):
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
Creating postgres (Lakebase) resources in the direct engine now retries transient server errors:
GET .../operations/...) is retried on transient errors (5xx, 408, inactivity timeouts) for every postgres resource. The create request was already accepted, so polling again cannot create a duplicate. A new helper,waitCreateOperation, wrapswaiter.Waitwithretry.IsTransientRead.POST /api/2.0/postgres/projects) is markedretrySafe, soDeploymentUnit.Createretries it on transient errors. The request is keyed by the caller-chosenproject_id, so a repeat cannot create a second project. If the failed request did create the project, the retry fails withALREADY_EXISTS, which is the same error the next deploy would have hit.Update and delete were already retried as a whole in
apply.go. Before this change, a create failure was returned without retrying.Judgment calls:
libs/retrypolicy: 2 retries, 15s apart.Why
Cloud integration runs on main hit 500s from the Lakebase API when creating a project, both on the create request and on the operation poll. These are 9 failures across 7 tests in 7 days on AWS and GCP. Most passed on the test runner's second attempt, so the errors are transient.
Dashboard:
Error: cannot create resources.postgres_projects.my_project: The request could not be processed. Please try again later and contact Databricks if the problem persists. [TraceId: [TRACE_ID]] (500 INTERNAL_ERROR)onPOST [DATABRICKS_URL]/api/2.0/postgres/projects?project_id=test-pg-proj-[UNIQUE_NAME]orGET [DATABRICKS_URL]/api/2.0/postgres/projects/test-pg-proj-[UNIQUE_NAME]/operations/...Affected tests: postgres_endpoints/basic/DMS=truepr=1Q)end=2026-11-12T13:40:00ZQ)last=7d, postgres_endpoints/recreate/DMS=truepr=1Q)end=2026-11-12T13:40:00ZQ)last=7d, postgres_endpoints/add_settings/DMS=pr=1Q)end=2026-11-12T13:40:00ZQ)last=7d, postgres_endpoints/update_autoscaling/DMS=pr=1Q)end=2026-11-12T13:40:00ZQ)last=7d, postgres_databases/update/DMS=pr=1Q)end=2026-11-12T13:40:00ZQ)last=7d, postgres_branches/update_ttl/DMS=truepr=1Q)end=2026-11-12T13:40:00ZQ)last=7d, postgres_projects/purge_on_delete/DMS=truepr=1Q)end=2026-11-12T13:40:00ZQ)last=7d.
Tests
bundle/resources/postgres_projects/create_transient_500injects one 500 on the create POST and one on the operation poll. With the fix, the deploy succeeds after two retries (shown in the output). Without the fix, it fails.bundle/resources/postgres_*acceptance tests and thebundle/direct/...unit tests pass.postgres_projects/basicandpostgres_endpoints/basicon every cloud.This pull request and its description were written by Isaac.