Skip to content

direct: use the postgres branch when its create request times out - #7602

Open
denik wants to merge 3 commits into
mainfrom
postgres-branch-create-504
Open

denik wants to merge 3 commits into
mainfrom
postgres-branch-create-504

Conversation

@denik

@denik denik commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Changes

When the request to create a postgres branch fails with a transient error (5xx, 408, inactivity timeout), the direct engine now checks whether the branch exists before giving up:

  • The branch exists: if it is READY, use it. If it is INIT, wait for READY (up to 10 minutes). In any other state (DELETED, ARCHIVED, ...), fail with the original error. Transient errors on this read are retried.
  • The branch does not exist: repeat the create. The error is marked retrySafe, so DeploymentUnit.Create retries it under the usual libs/retry policy.
  • replace_existing: true: repeat the create directly. A branch found at this point may have existed before the request, and repeating a replace is safe.

Follow-up to #7557, which retried transient errors while polling create operations and on the project create request. Branches were left out there because they weren't failing yet.

Judgment calls:

  • A branch that is found is used without checking that this request created it. Without replace_existing, a branch that already existed would have made the request fail with ALREADY_EXISTS rather than time out, so finding it after a timeout almost always means this request created it.
  • If reading the branch fails, the original create error is returned, and the read error is logged as a warning.
  • Known gap: the read right after the 504 may say the branch is missing while the backend is still creating it. The repeated create then fails with ALREADY_EXISTS, which is no worse than before this change. Handling that would mean adopting a branch on ALREADY_EXISTS, which can't tell our branch from one that existed before, so I left it out.
  • The INIT wait isn't covered by a local test, because the testserver creates branches as READY.
  • Only branches are changed. Projects have no ready state to wait for, and their create request was already made retry-safe in #7557.

Why

The Lakebase API sometimes answers a branch create with a 504 that says the branch may still be created. The deploy failed even when the branch had been created, and dependent resources failed with it.

Dashboard:

error snippet
error/3325 Error: cannot create resources.postgres_branches.main: The create request timed out and may still be completing. Check whether the resource was created before retrying. (504 DEADLINE_EXCEEDED) on POST [DATABRICKS_URL]/api/2.0/postgres/projects/test-pg-proj-[UNIQUE_NAME]/branches?branch_id=main

Tests: postgres_databases/update/DMS=true, postgres_roles/replace_existing/DMS=.

Tests

  • New local acceptance test bundle/resources/postgres_branches/create_504 injects a 504 on the branch create in two ways: once after the branch was created, and once without creating it. The deploy succeeds both times; the request log shows the branch being used in the first case and the create being repeated in the second. Without the fix, both deploys fail.
  • The existing local bundle/resources/postgres_* acceptance tests and the bundle/direct/... unit tests pass.
  • The last commit asks CI to run postgres_branches and postgres_roles/replace_existing on every cloud.

This pull request and its description were written by Isaac.

Creating a postgres branch sometimes fails with 504 DEADLINE_EXCEEDED, "The
create request timed out and may still be completing. Check whether the
resource was created before retrying." The deploy failed even when the
branch had been created.

On a transient create error, read the branch: if it exists, wait until it is
READY and use it; if it does not, repeat the create (retrySafe). With
replace_existing the request is repeated directly, since a branch found then
may predate it.

cloud: TestAccept/bundle/resources/postgres_branches TestAccept/bundle/resources/postgres_roles/replace_existing

Co-authored-by: Isaac <no-reply@databricks.com>
@denik denik added the ai-auto PRs made by AI loops & factories, label Nov 13, 2026
@github-actions github-actions Bot added the DABs DABs related issues label Nov 13, 2026
denik and others added 2 commits November 13, 2026 13:28
cloud: TestAccept/bundle/resources/postgres_branches TestAccept/bundle/resources/postgres_roles/replace_existing

Co-authored-by: Isaac <no-reply@databricks.com>
Addresses review: a found branch in DELETED/ARCHIVED never becomes READY, so
fail at once instead of polling for 10 minutes; retry transient errors on the
GET used to check the branch.

cloud: TestAccept/bundle/resources/postgres_branches TestAccept/bundle/resources/postgres_roles/replace_existing

Co-authored-by: Isaac <no-reply@databricks.com>
@github-actions

Copy link
Copy Markdown
Contributor

Commit: 50c2d4132 | Workflow run

Regression test report

Selected 1 changed tests (limit=20, 0 not selected)

24 tests: regression=2 unreleased=0 coverage=22 failing=0
test branch main (5ed1897) latest (v1.26.0)
TestAccept/bundle/resources/postgres_branches/basic/DMS= ✅ ✅ ✅
TestAccept/bundle/resources/postgres_branches/basic/DMS=true ✅ ✅ ✅
TestAccept/bundle/resources/postgres_branches/create_504/DMS= ✅ ❌ ❌
TestAccept/bundle/resources/postgres_branches/create_504/DMS=true ✅ ❌ ❌
TestAccept/bundle/resources/postgres_branches/purge_on_delete/DMS= ✅ ✅ ✅
TestAccept/bundle/resources/postgres_branches/purge_on_delete/DMS=true ✅ ✅ ✅
TestAccept/bundle/resources/postgres_branches/purge_on_delete_transitions/DMS= ✅ ✅ ✅
TestAccept/bundle/resources/postgres_branches/purge_on_delete_transitions/DMS=true ✅ ✅ ✅
TestAccept/bundle/resources/postgres_branches/recreate/DMS= ✅ ✅ ✅
TestAccept/bundle/resources/postgres_branches/recreate/DMS=true ✅ ✅ ✅
TestAccept/bundle/resources/postgres_branches/recreate_source_branch_time/DMS= ✅ ✅ ✅
TestAccept/bundle/resources/postgres_branches/recreate_source_branch_time/DMS=true ✅ ✅ ✅
TestAccept/bundle/resources/postgres_branches/replace_existing/DMS= ✅ ✅ ✅
TestAccept/bundle/resources/postgres_branches/replace_existing/DMS=true ✅ ✅ ✅
TestAccept/bundle/resources/postgres_branches/update_expire_time/DMS= ✅ ✅ ✅
TestAccept/bundle/resources/postgres_branches/update_expire_time/DMS=true ✅ ✅ ✅
TestAccept/bundle/resources/postgres_branches/update_protected/DMS= ✅ ✅ ✅
TestAccept/bundle/resources/postgres_branches/update_protected/DMS=true ✅ ✅ ✅
TestAccept/bundle/resources/postgres_branches/update_ttl/DMS= ✅ ✅ ✅
TestAccept/bundle/resources/postgres_branches/update_ttl/DMS=true ✅ ✅ ✅
TestAccept/bundle/resources/postgres_branches/without_branch_id/DMS= ✅ ✅ ✅
TestAccept/bundle/resources/postgres_branches/without_branch_id/DMS=true ✅ ✅ ✅
TestAccept/bundle/resources/postgres_roles/replace_existing/DMS= ✅ ✅ ✅
TestAccept/bundle/resources/postgres_roles/replace_existing/DMS=true ✅ ✅ ✅
TestAccept/bundle/resources/postgres_branches/create_504/DMS=: branch ✅ | main (5ed1897) ❌ | latest (v1.26.0) ❌

main (5ed1897):

acceptance_test.go:1374: Diff:
        --- bundle/resources/postgres_branches/create_504/output.txt
        +++ /tmp/TestAcceptbundleresourcespostgres_branchescreate_504DMS=3910832053/002/output.txt
        @@ -4,27 +4,13 @@
         >>> [CLI] bundle deploy
         Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files...
         Created postgres_projects.my_project
        -Warn: deploying resources.postgres_branches.dev: the branch was created despite the failed create request: Fault injected by test.
        -Created postgres_branches.dev
        -Files: 0 uploaded, 0 deleted
        -Resources: 2 created, 0 changed, 0 deleted, 0 unchanged
        +Error: cannot create resources.postgres_branches.dev: Fault injected by test. (504 DEADLINE_EXCEEDED)
         
        ->>> print_requests.py --get //postgres/projects/test-pg-proj/branches
        -POST /api/2.0/postgres/projects/test-pg-proj/branches
        -GET /api/2.0/postgres/projects/test-pg-proj/branches/dev
        +Endpoint: POST [DATABRICKS_URL]/api/2.0/postgres/projects/test-pg-proj/branches?branch_id=dev
        +HTTP Status: 504 Gateway Timeout
        +API error_code: DEADLINE_EXCEEDED
        +API message: Fault injected by test.
         
        -=== The create request times out and the branch was not created: deploy repeats the create
        -
        ->>> [CLI] bundle deploy
        -Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files...
        -Created postgres_projects.my_project
        -Warn: deploying resources.postgres_branches.dev: retrying after 504 Gateway Timeout from POST /api/2.0/postgres/projects/test-pg-proj/branches
        -Created postgres_branches.dev
         Files: 0 uploaded, 0 deleted
        -Resources: 2 created, 0 changed, 0 deleted, 0 unchanged
         
        ->>> print_requests.py --get //postgres/projects/test-pg-proj/branches
        -POST /api/2.0/postgres/projects/test-pg-proj/branches
        -GET /api/2.0/postgres/projects/test-pg-proj/branches/dev
        -POST /api/2.0/postgres/projects/test-pg-proj/branches
        -GET /api/2.0/postgres/projects/test-pg-proj/branches/dev/operations/[UUID]
        +Exit code: 1

latest (v1.26.0): same output as main (5ed1897)

TestAccept/bundle/resources/postgres_branches/create_504/DMS=true: branch ✅ | main (5ed1897) ❌ | latest (v1.26.0) ❌

main (5ed1897):

acceptance_test.go:1374: Diff:
        --- bundle/resources/postgres_branches/create_504/output.txt
        +++ /tmp/TestAcceptbundleresourcespostgres_branchescreate_504DMS=tru2354854587/002/output.txt
        @@ -4,27 +4,13 @@
         >>> [CLI] bundle deploy
         Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files...
         Created postgres_projects.my_project
        -Warn: deploying resources.postgres_branches.dev: the branch was created despite the failed create request: Fault injected by test.
        -Created postgres_branches.dev
        -Files: 0 uploaded, 0 deleted
        -Resources: 2 created, 0 changed, 0 deleted, 0 unchanged
        +Error: cannot create resources.postgres_branches.dev: Fault injected by test. (504 DEADLINE_EXCEEDED)
         
        ->>> print_requests.py --get //postgres/projects/test-pg-proj/branches
        -POST /api/2.0/postgres/projects/test-pg-proj/branches
        -GET /api/2.0/postgres/projects/test-pg-proj/branches/dev
        +Endpoint: POST [DATABRICKS_URL]/api/2.0/postgres/projects/test-pg-proj/branches?branch_id=dev
        +HTTP Status: 504 Gateway Timeout
        +API error_code: DEADLINE_EXCEEDED
        +API message: Fault injected by test.
         
        -=== The create request times out and the branch was not created: deploy repeats the create
        -
        ->>> [CLI] bundle deploy
        -Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files...
        -Created postgres_projects.my_project
        -Warn: deploying resources.postgres_branches.dev: retrying after 504 Gateway Timeout from POST /api/2.0/postgres/projects/test-pg-proj/branches
        -Created postgres_branches.dev
         Files: 0 uploaded, 0 deleted
        -Resources: 2 created, 0 changed, 0 deleted, 0 unchanged
         
        ->>> print_requests.py --get //postgres/projects/test-pg-proj/branches
        -POST /api/2.0/postgres/projects/test-pg-proj/branches
        -GET /api/2.0/postgres/projects/test-pg-proj/branches/dev
        -POST /api/2.0/postgres/projects/test-pg-proj/branches
        -GET /api/2.0/postgres/projects/test-pg-proj/branches/dev/operations/[UUID]
        +Exit code: 1

latest (v1.26.0): same output as main (5ed1897)

@eng-dev-ecosystem-bot

Copy link
Copy Markdown
Collaborator

✅ Integration test report

Commit: 50c2d41

Run: 43434719002

Env 🔄​flaky 💚​RECOVERED ✅​pass 🙈​skip Time
💚​ aws linux-2core-8gb 1 297 18 13:31
💚​ aws-windows-latest-4core-16gb 1 299 16 9:27
💚​ azure linux-2core-8gb 1 296 18 13:47
🔄​ azure-windows-latest-4core-16gb 1 1 297 16 11:26
💚​ gcp linux-2core-8gb 1 297 18 12:46
💚​ gcp-windows-latest-4core-16gb 1 299 16 10:08
Test Name aws linux-2core-8gb aws-windows-latest-4core-16gb azure linux-2core-8gb azure-windows-latest-4core-16gb gcp linux-2core-8gb gcp-windows-latest-4core-16gb
💚​ TestAccept 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
🔄​ TestSyncIncrementalFileOverwritesFolder ✅​p ✅​p ✅​p 🔄​f ✅​p ✅​p
Top 32 slowest tests (at least 2 minutes):
duration env testname
3:55 azure linux-2core-8gb TestAccept
3:53 aws linux-2core-8gb TestAccept
3:45 gcp linux-2core-8gb TestAccept
3:30 aws-windows-latest-4core-16gb TestAccept/bundle/resources/postgres_branches/update_protected/DMS=
3:22 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/postgres_branches/purge_on_delete_transitions/DMS=true
3:17 gcp-windows-latest-4core-16gb TestAccept
3:17 azure-windows-latest-4core-16gb TestAccept/bundle/resources/postgres_branches/update_protected/DMS=
3:16 azure-windows-latest-4core-16gb TestAccept
3:04 aws linux-2core-8gb TestAccept/bundle/resources/postgres_branches/update_protected/DMS=
2:59 aws linux-2core-8gb TestAccept/bundle/resources/postgres_roles/replace_existing/DMS=true
2:58 azure linux-2core-8gb TestAccept/bundle/resources/postgres_branches/recreate/DMS=
2:53 azure-windows-latest-4core-16gb TestAccept/bundle/resources/postgres_branches/update_protected/DMS=true
2:40 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/postgres_branches/update_protected/DMS=true
2:38 aws-windows-latest-4core-16gb TestAccept
2:36 azure-windows-latest-4core-16gb TestAccept/bundle/resources/postgres_branches/purge_on_delete_transitions/DMS=true
2:33 azure-windows-latest-4core-16gb TestAccept/bundle/resources/postgres_branches/purge_on_delete_transitions/DMS=
2:33 gcp linux-2core-8gb TestAccept/bundle/resources/postgres_branches/update_protected/DMS=true
2:30 aws linux-2core-8gb TestAccept/bundle/resources/postgres_branches/update_protected/DMS=true
2:30 aws linux-2core-8gb TestAccept/bundle/resources/postgres_branches/recreate/DMS=true
2:27 gcp linux-2core-8gb TestAccept/bundle/resources/postgres_branches/purge_on_delete_transitions/DMS=true
2:24 gcp linux-2core-8gb TestFilerWorkspaceFilesExtensionsReadDir
2:20 gcp linux-2core-8gb TestAccept/bundle/resources/postgres_branches/recreate/DMS=
2:19 azure linux-2core-8gb TestAccept/bundle/resources/postgres_branches/without_branch_id/DMS=true
2:19 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/postgres_branches/purge_on_delete_transitions/DMS=
2:13 azure linux-2core-8gb TestAccept/bundle/resources/postgres_roles/replace_existing/DMS=
2:12 aws-windows-latest-4core-16gb TestAccept/bundle/resources/postgres_branches/purge_on_delete_transitions/DMS=
2:12 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/postgres_branches/update_ttl/DMS=
2:11 azure-windows-latest-4core-16gb TestAccept/bundle/resources/postgres_branches/recreate/DMS=true
2:09 gcp linux-2core-8gb TestAccept/bundle/resources/postgres_branches/purge_on_delete_transitions/DMS=
2:05 gcp-windows-latest-4core-16gb TestWorkspaceFilesExtensions_ExportFormatIsPreserved/jupyter_scala
2:02 aws-windows-latest-4core-16gb TestAccept/bundle/resources/postgres_roles/replace_existing/DMS=true
2:01 azure linux-2core-8gb TestAccept/bundle/resources/postgres_roles/replace_existing/DMS=true

@denik
denik marked this pull request as ready for review November 13, 2026 14:03
@denik
denik requested review from a team as code owners November 13, 2026 14:03

@Sankalp-Mittal Sankalp-Mittal left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 AI review from review-factory: 3 independent reviewer models (Claude Sonnet 5.5, Claude Opus 5.5, Claude Fable 5.2) with cross-verification. Only findings that reviewers agreed on are posted.

Postgres branch create now recovers from transient errors by reading the branch; reviewers flagged untested paths and an unsuppressed warning.

Findings: 2 major, 3 minor, 1 nit.


name := config.Parent + "/branches/" + config.BranchId
deadline := time.Now().Add(branchReadyTimeout)
for {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

major: Most of recoverFailedCreate is untested: INIT wait, non-INIT state, timeout, read failure, replace_existing

create_504 exercises two outcomes only: the branch is found READY on the first read, and the branch is missing so the create is repeated. These paths of recoverFailedCreate have no coverage:

  • the INIT poll loop (about lines 187-197), which the PR description presents as a main new behavior and which a "may still be completing" 504 is likely to hit;
  • the non-INIT/non-READY state error (line 188) and the 10-minute timeout (line 191);
  • a read that fails with something other than not-found (lines 172-175);
  • replace_existing: true (lines 159-161), which repeats the create without reading and is the documented way to manage the implicit production branch;
  • a non-transient create error (no GET issued).

The PR says the INIT path can't be tested because the testserver always creates branches as READY (libs/testserver/postgres.go:347-349). That rules out an acceptance test for INIT, but not a unit test. Other resources in this package test DoCreate against mocks.NewMockWorkspaceClient(t) (genie_space_test.go:60, feature_spec_test.go:44; the mock exposes GetMockPostgresAPI()), and job_run_test.go:39-66 drives a wait with testserver.New(t) and a custom server.Handle. There is no postgres_branch_test.go. A table-driven test could cover: GET returns INIT then READY; GET returns an unexpected state (the error carries the original create error); GET fails with a non-missing error (the original error is returned); replace_existing: true (no GET, RetrySafeError); a non-transient create error (no GET).

Two literals block such a test: the poll interval time.After(5 * time.Second) (:196) ignores DATABRICKS_BUNDLE_RETRY_INTERVAL_MS, and the deadline is the const branchReadyTimeout (:147). An INIT test would sleep 5s and the timeout path can't be reached. Make them fields or package vars a test can shrink.

Two of the gaps can also be closed with acceptance scenarios and no testserver changes, using the existing fault harness (acceptance/bin/fault.py supports --after-handler and arbitrary method/path patterns):

  • replace_existing: true with fault.py --after-handler "POST /api/2.0/postgres/projects/*/branches" 504 0 1 DEADLINE_EXCEEDED should log POST, POST, operation GET, and no branch GET.
  • A failed read: a GET fault such as fault.py "GET /api/2.0/postgres/projects/*/branches/dev" 500 0 3 after a 504 on the POST should show the checking whether ... was created warning and the original 504 error, with no repeated POST.

These tests don't duplicate create_504: that covers the READY and missing paths, and these cover paths it can't reach.

Agreed by 3 reviewer model(s).

state = branch.Status.CurrentState
}
if state == postgres.BranchStatusStateReady {
log.Warnf(ctx, "the branch was created despite the failed create request: %s", createErr)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

major: The new adoption warning is not suppressed, so the cloud tests the PR targets still fail when the branch is adopted

When recovery adopts the branch (found READY, or READY after the INIT wait), it logs Warn: deploying resources.postgres_branches.<key>: the branch was created despite the failed create request: <api message>. Acceptance output only drops warnings matching ^Warn: (.*: )?retrying( after |$), and this line does not match it.

The two tests the description names as affected (postgres_roles/replace_existing, postgres_databases/update) run trace $CLI bundle deploy and pin the result in output.txt. On a real 504 the deploy now succeeds, but the extra Warn: line makes the golden comparison fail, so those tests stay red on exactly the runs this PR targets. Only the missing-branch and replace_existing paths are safe, because their only warning is the suppressed retrying after ....

In postgres_databases/update, the recovery GET .../branches/main would also land in out.requests.create.json, since that recording uses --get '//postgres' and excludes only /operations/.

Suggested fix: either log the adoption below Warn, or add a second root SuppressLines pattern for it and opt create_504 out of it the same way it opts out of warn_retry.

Agreed by 3 reviewer model(s).

if branch.Status != nil {
state = branch.Status.CurrentState
}
if state == postgres.BranchStatusStateReady {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor: A pre-existing READY branch can be adopted after any transient error, not only a handler timeout

The PR argues that without replace_existing a pre-existing branch would have made the POST fail with ALREADY_EXISTS, so a branch found after the failure must have been created by this request. That holds only if the request reached the backend's existence check. The gate is retry.IsTransient, which is true for any 5xx or 408 the SDK did not retry and for the client-side inactivity timeout. That includes a 502/503 from an edge proxy that never forwarded the request and a backend 500 raised before the ID-uniqueness check.

In those cases a READY branch with the same ID that already existed (created by hand, by another bundle, left from an earlier deploy whose local state was lost, or the implicit production branch every project has) is adopted. The deploy prints Created, the configured spec was never sent, the warning wrongly says the branch "was created despite the failed create request", the branch is recorded in state with the bundle's config, and a later bundle destroy deletes a branch the bundle did not create. replace_existing is the explicit opt-in for taking over an existing branch, and this path bypasses it. Before this change the same sequence failed the deploy.

The window is narrow (a 5xx must coincide with an ID collision), but the outcome is a silent takeover. A cheap guard: record time.Now() before CreateBranch and adopt only when branch.CreateTime is not clearly earlier (allowing a margin for clock skew), otherwise return createErr unchanged. makePostgresBranchRemote (:87-102) shows CreateTime is available on the response. If you keep the current behavior, the warning could name the branch and say it may predate the request.

Agreed by 3 reviewer model(s).

branch, err := retry.With(ctx, retry.IsTransientRead, func() (*postgres.Branch, error) {
return r.client.Postgres.GetBranch(ctx, postgres.GetBranchRequest{Name: name})
})
if apierr.IsMissing(err) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor: In the "may still be completing" case, the repeated create can end in ALREADY_EXISTS

The backend's 504 says the request "may still be completing", yet the branch is read right away (:166) with no pause. If the backend has not committed the branch yet, the read returns 404. The code then returns retrySafe(createErr), and DeploymentUnit.Create repeats the POST about 15s later. If the first request has landed by then, that POST returns 409 ALREADY_EXISTS, which is not transient, so the deploy fails. That is the failure this PR targets, now with a different message.

The PR lists this as a known gap. But its own argument (the first attempt got a 504, not a 409, so no branch existed beforehand) applies equally to a 409 that follows our own transient failure in the same Create. Options:

  • Re-check after a short pause before returning retrySafe.
  • Have recoverFailedCreate repeat the create itself, so it can treat ALREADY_EXISTS after its own 504 like the READY case.

Without one of these, the fix only helps when the branch already exists when the 504 arrives, which is the case the --after-handler test models.

Agreed by 3 reviewer model(s).


name := config.Parent + "/branches/" + config.BranchId
deadline := time.Now().Add(branchReadyTimeout)
for {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor: Up to 10 minutes of polling inside DoCreate, before state is saved and outside the max-wait cap

bundle/direct/dresources/README.md:44 says to implement WaitAfterCreate instead of polling inline inside DoCreate, because inline polling bypasses state persistence timing (apply.go saves state before WaitAfterCreate runs). The new loop polls for up to branchReadyTimeout (10 minutes) inside DoCreate.

Concrete effects:

  • At the timeout (line 191) and on a cancelled context (line 195) the code already knows the branch exists under name, yet returns an error, so nothing is written to state. If the 10-minute loop times out or the user presses Ctrl-C, the next deploy fails with ALREADY_EXISTS.
  • DATABRICKS_BUNDLE_RESOURCE_MAX_WAIT does not bound this wait, because waitCapped wraps only WaitAfterCreate.

The existing success path also waits inline (waitCreateOperation at :137), so this is not new in kind, and moving it could be a follow-up. But here the ID is known as soon as the read succeeds: returning the branch once it is found and waiting for READY in a WaitAfterCreate would persist state first, respect the cap, and reuse the framework's transient-read retry instead of a second hand-written loop. If you keep it inline, a short comment on why would help the next reader.

Agreed by 3 reviewer model(s).

if apierr.IsMissing(err) {
return "", nil, retrySafe(createErr)
}
if err != nil {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: A cancelled context during the recovery read is logged as a failed existence check and replaced by the create error

If ctx is cancelled while the read is inside retry.With, it returns ctx.Err(). That is not apierr.IsMissing, so it falls into the generic branch, logs checking whether ... was created: context canceled at warn level, and returns the original 504 instead of the cancellation. The deploy still stops, but the message is misleading. Check ctx.Err() first, or let that error pass through unwrapped as the select in the INIT loop does.

Agreed by 3 reviewer model(s).

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ai-auto PRs made by AI loops & factories, DABs DABs related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants