Repository navigation
Conversation
Creating a postgres branch sometimes fails with 504 DEADLINE_EXCEEDED, "The create request timed out and may still be completing. Check whether the resource was created before retrying." The deploy failed even when the branch had been created. On a transient create error, read the branch: if it exists, wait until it is READY and use it; if it does not, repeat the create (retrySafe). With replace_existing the request is repeated directly, since a branch found then may predate it. cloud: TestAccept/bundle/resources/postgres_branches TestAccept/bundle/resources/postgres_roles/replace_existing Co-authored-by: Isaac <no-reply@databricks.com>
cloud: TestAccept/bundle/resources/postgres_branches TestAccept/bundle/resources/postgres_roles/replace_existing Co-authored-by: Isaac <no-reply@databricks.com>
Addresses review: a found branch in DELETED/ARCHIVED never becomes READY, so fail at once instead of polling for 10 minutes; retry transient errors on the GET used to check the branch. cloud: TestAccept/bundle/resources/postgres_branches TestAccept/bundle/resources/postgres_roles/replace_existing Co-authored-by: Isaac <no-reply@databricks.com>
|
Commit: Regression test reportSelected 1 changed tests (limit=20, 0 not selected) 24 tests: regression=2 unreleased=0 coverage=22 failing=0
TestAccept/bundle/resources/postgres_branches/create_504/DMS=: branch ✅ | main (5ed1897) ❌ | latest (v1.26.0) ❌main (5ed1897): latest (v1.26.0): same output as main (5ed1897) TestAccept/bundle/resources/postgres_branches/create_504/DMS=true: branch ✅ | main (5ed1897) ❌ | latest (v1.26.0) ❌main (5ed1897): latest (v1.26.0): same output as main (5ed1897) |
✅ Integration test reportCommit: 50c2d41
Top 32 slowest tests (at least 2 minutes):
|
Sankalp-Mittal
left a comment
There was a problem hiding this comment.
🤖 AI review from review-factory: 3 independent reviewer models (Claude Sonnet 5.5, Claude Opus 5.5, Claude Fable 5.2) with cross-verification. Only findings that reviewers agreed on are posted.
Postgres branch create now recovers from transient errors by reading the branch; reviewers flagged untested paths and an unsuppressed warning.
Findings: 2 major, 3 minor, 1 nit.
|
|
||
| name := config.Parent + "/branches/" + config.BranchId | ||
| deadline := time.Now().Add(branchReadyTimeout) | ||
| for { |
There was a problem hiding this comment.
major: Most of recoverFailedCreate is untested: INIT wait, non-INIT state, timeout, read failure, replace_existing
create_504 exercises two outcomes only: the branch is found READY on the first read, and the branch is missing so the create is repeated. These paths of recoverFailedCreate have no coverage:
- the
INITpoll loop (about lines 187-197), which the PR description presents as a main new behavior and which a "may still be completing" 504 is likely to hit; - the non-
INIT/non-READYstate error (line 188) and the 10-minute timeout (line 191); - a read that fails with something other than not-found (lines 172-175);
replace_existing: true(lines 159-161), which repeats the create without reading and is the documented way to manage the implicitproductionbranch;- a non-transient create error (no GET issued).
The PR says the INIT path can't be tested because the testserver always creates branches as READY (libs/testserver/postgres.go:347-349). That rules out an acceptance test for INIT, but not a unit test. Other resources in this package test DoCreate against mocks.NewMockWorkspaceClient(t) (genie_space_test.go:60, feature_spec_test.go:44; the mock exposes GetMockPostgresAPI()), and job_run_test.go:39-66 drives a wait with testserver.New(t) and a custom server.Handle. There is no postgres_branch_test.go. A table-driven test could cover: GET returns INIT then READY; GET returns an unexpected state (the error carries the original create error); GET fails with a non-missing error (the original error is returned); replace_existing: true (no GET, RetrySafeError); a non-transient create error (no GET).
Two literals block such a test: the poll interval time.After(5 * time.Second) (:196) ignores DATABRICKS_BUNDLE_RETRY_INTERVAL_MS, and the deadline is the const branchReadyTimeout (:147). An INIT test would sleep 5s and the timeout path can't be reached. Make them fields or package vars a test can shrink.
Two of the gaps can also be closed with acceptance scenarios and no testserver changes, using the existing fault harness (acceptance/bin/fault.py supports --after-handler and arbitrary method/path patterns):
replace_existing: truewithfault.py --after-handler "POST /api/2.0/postgres/projects/*/branches" 504 0 1 DEADLINE_EXCEEDEDshould logPOST,POST, operationGET, and no branchGET.- A failed read: a GET fault such as
fault.py "GET /api/2.0/postgres/projects/*/branches/dev" 500 0 3after a 504 on the POST should show thechecking whether ... was createdwarning and the original 504 error, with no repeated POST.
These tests don't duplicate create_504: that covers the READY and missing paths, and these cover paths it can't reach.
Agreed by 3 reviewer model(s).
| state = branch.Status.CurrentState | ||
| } | ||
| if state == postgres.BranchStatusStateReady { | ||
| log.Warnf(ctx, "the branch was created despite the failed create request: %s", createErr) |
There was a problem hiding this comment.
major: The new adoption warning is not suppressed, so the cloud tests the PR targets still fail when the branch is adopted
When recovery adopts the branch (found READY, or READY after the INIT wait), it logs Warn: deploying resources.postgres_branches.<key>: the branch was created despite the failed create request: <api message>. Acceptance output only drops warnings matching ^Warn: (.*: )?retrying( after |$), and this line does not match it.
The two tests the description names as affected (postgres_roles/replace_existing, postgres_databases/update) run trace $CLI bundle deploy and pin the result in output.txt. On a real 504 the deploy now succeeds, but the extra Warn: line makes the golden comparison fail, so those tests stay red on exactly the runs this PR targets. Only the missing-branch and replace_existing paths are safe, because their only warning is the suppressed retrying after ....
In postgres_databases/update, the recovery GET .../branches/main would also land in out.requests.create.json, since that recording uses --get '//postgres' and excludes only /operations/.
Suggested fix: either log the adoption below Warn, or add a second root SuppressLines pattern for it and opt create_504 out of it the same way it opts out of warn_retry.
Agreed by 3 reviewer model(s).
| if branch.Status != nil { | ||
| state = branch.Status.CurrentState | ||
| } | ||
| if state == postgres.BranchStatusStateReady { |
There was a problem hiding this comment.
minor: A pre-existing READY branch can be adopted after any transient error, not only a handler timeout
The PR argues that without replace_existing a pre-existing branch would have made the POST fail with ALREADY_EXISTS, so a branch found after the failure must have been created by this request. That holds only if the request reached the backend's existence check. The gate is retry.IsTransient, which is true for any 5xx or 408 the SDK did not retry and for the client-side inactivity timeout. That includes a 502/503 from an edge proxy that never forwarded the request and a backend 500 raised before the ID-uniqueness check.
In those cases a READY branch with the same ID that already existed (created by hand, by another bundle, left from an earlier deploy whose local state was lost, or the implicit production branch every project has) is adopted. The deploy prints Created, the configured spec was never sent, the warning wrongly says the branch "was created despite the failed create request", the branch is recorded in state with the bundle's config, and a later bundle destroy deletes a branch the bundle did not create. replace_existing is the explicit opt-in for taking over an existing branch, and this path bypasses it. Before this change the same sequence failed the deploy.
The window is narrow (a 5xx must coincide with an ID collision), but the outcome is a silent takeover. A cheap guard: record time.Now() before CreateBranch and adopt only when branch.CreateTime is not clearly earlier (allowing a margin for clock skew), otherwise return createErr unchanged. makePostgresBranchRemote (:87-102) shows CreateTime is available on the response. If you keep the current behavior, the warning could name the branch and say it may predate the request.
Agreed by 3 reviewer model(s).
| branch, err := retry.With(ctx, retry.IsTransientRead, func() (*postgres.Branch, error) { | ||
| return r.client.Postgres.GetBranch(ctx, postgres.GetBranchRequest{Name: name}) | ||
| }) | ||
| if apierr.IsMissing(err) { |
There was a problem hiding this comment.
minor: In the "may still be completing" case, the repeated create can end in ALREADY_EXISTS
The backend's 504 says the request "may still be completing", yet the branch is read right away (:166) with no pause. If the backend has not committed the branch yet, the read returns 404. The code then returns retrySafe(createErr), and DeploymentUnit.Create repeats the POST about 15s later. If the first request has landed by then, that POST returns 409 ALREADY_EXISTS, which is not transient, so the deploy fails. That is the failure this PR targets, now with a different message.
The PR lists this as a known gap. But its own argument (the first attempt got a 504, not a 409, so no branch existed beforehand) applies equally to a 409 that follows our own transient failure in the same Create. Options:
- Re-check after a short pause before returning
retrySafe. - Have
recoverFailedCreaterepeat the create itself, so it can treatALREADY_EXISTSafter its own 504 like the READY case.
Without one of these, the fix only helps when the branch already exists when the 504 arrives, which is the case the --after-handler test models.
Agreed by 3 reviewer model(s).
|
|
||
| name := config.Parent + "/branches/" + config.BranchId | ||
| deadline := time.Now().Add(branchReadyTimeout) | ||
| for { |
There was a problem hiding this comment.
minor: Up to 10 minutes of polling inside DoCreate, before state is saved and outside the max-wait cap
bundle/direct/dresources/README.md:44 says to implement WaitAfterCreate instead of polling inline inside DoCreate, because inline polling bypasses state persistence timing (apply.go saves state before WaitAfterCreate runs). The new loop polls for up to branchReadyTimeout (10 minutes) inside DoCreate.
Concrete effects:
- At the timeout (line 191) and on a cancelled context (line 195) the code already knows the branch exists under
name, yet returns an error, so nothing is written to state. If the 10-minute loop times out or the user presses Ctrl-C, the next deploy fails withALREADY_EXISTS. DATABRICKS_BUNDLE_RESOURCE_MAX_WAITdoes not bound this wait, becausewaitCappedwraps onlyWaitAfterCreate.
The existing success path also waits inline (waitCreateOperation at :137), so this is not new in kind, and moving it could be a follow-up. But here the ID is known as soon as the read succeeds: returning the branch once it is found and waiting for READY in a WaitAfterCreate would persist state first, respect the cap, and reuse the framework's transient-read retry instead of a second hand-written loop. If you keep it inline, a short comment on why would help the next reader.
Agreed by 3 reviewer model(s).
| if apierr.IsMissing(err) { | ||
| return "", nil, retrySafe(createErr) | ||
| } | ||
| if err != nil { |
There was a problem hiding this comment.
nit: A cancelled context during the recovery read is logged as a failed existence check and replaced by the create error
If ctx is cancelled while the read is inside retry.With, it returns ctx.Err(). That is not apierr.IsMissing, so it falls into the generic branch, logs checking whether ... was created: context canceled at warn level, and returns the original 504 instead of the cancellation. The deploy still stops, but the message is misleading. Check ctx.Err() first, or let that error pass through unwrapped as the select in the INIT loop does.
Agreed by 3 reviewer model(s).
Changes
When the request to create a postgres branch fails with a transient error (5xx, 408, inactivity timeout), the direct engine now checks whether the branch exists before giving up:
READY, use it. If it isINIT, wait forREADY(up to 10 minutes). In any other state (DELETED,ARCHIVED, ...), fail with the original error. Transient errors on this read are retried.retrySafe, soDeploymentUnit.Createretries it under the usuallibs/retrypolicy.replace_existing: true: repeat the create directly. A branch found at this point may have existed before the request, and repeating a replace is safe.Follow-up to #7557, which retried transient errors while polling create operations and on the project create request. Branches were left out there because they weren't failing yet.
Judgment calls:
replace_existing, a branch that already existed would have made the request fail withALREADY_EXISTSrather than time out, so finding it after a timeout almost always means this request created it.ALREADY_EXISTS, which is no worse than before this change. Handling that would mean adopting a branch onALREADY_EXISTS, which can't tell our branch from one that existed before, so I left it out.INITwait isn't covered by a local test, because the testserver creates branches asREADY.Why
The Lakebase API sometimes answers a branch create with a 504 that says the branch may still be created. The deploy failed even when the branch had been created, and dependent resources failed with it.
Dashboard:
Error: cannot create resources.postgres_branches.main: The create request timed out and may still be completing. Check whether the resource was created before retrying. (504 DEADLINE_EXCEEDED)onPOST [DATABRICKS_URL]/api/2.0/postgres/projects/test-pg-proj-[UNIQUE_NAME]/branches?branch_id=mainTests: postgres_databases/update/DMS=true, postgres_roles/replace_existing/DMS=.
Tests
bundle/resources/postgres_branches/create_504injects a 504 on the branch create in two ways: once after the branch was created, and once without creating it. The deploy succeeds both times; the request log shows the branch being used in the first case and the create being repeated in the second. Without the fix, both deploys fail.bundle/resources/postgres_*acceptance tests and thebundle/direct/...unit tests pass.postgres_branchesandpostgres_roles/replace_existingon every cloud.This pull request and its description were written by Isaac.