Repository navigation
direct: use the postgres branch when its create request times out #7602
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| * Use the Lakebase branch when its create request times out but the branch was created, instead of failing the deploy, in the direct deployment engine. ([#7602](https://github.com/databricks/cli/pull/7602)) |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,18 @@ | ||
| bundle: | ||
| name: test-bundle | ||
|
|
||
| sync: | ||
| paths: [] | ||
|
|
||
| resources: | ||
| postgres_projects: | ||
| my_project: | ||
| project_id: test-pg-proj | ||
| display_name: "Test Postgres Project" | ||
| pg_version: 16 | ||
|
|
||
| postgres_branches: | ||
| dev: | ||
| parent: ${resources.postgres_projects.my_project.id} | ||
| branch_id: dev | ||
| no_expiry: true |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,3 @@ | ||
| Cloud = false | ||
| EnvMatrix.DMS = ["", "true"] | ||
| EnvMatrix.READPLAN = [] |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,30 @@ | ||
|
|
||
| === The create request times out but the branch was created: deploy uses the branch | ||
|
|
||
| >>> [CLI] bundle deploy | ||
| Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files... | ||
| Created postgres_projects.my_project | ||
| Warn: deploying resources.postgres_branches.dev: the branch was created despite the failed create request: Fault injected by test. | ||
| Created postgres_branches.dev | ||
| Files: 0 uploaded, 0 deleted | ||
| Resources: 2 created, 0 changed, 0 deleted, 0 unchanged | ||
|
|
||
| >>> print_requests.py --get //postgres/projects/test-pg-proj/branches | ||
| POST /api/2.0/postgres/projects/test-pg-proj/branches | ||
| GET /api/2.0/postgres/projects/test-pg-proj/branches/dev | ||
|
|
||
| === The create request times out and the branch was not created: deploy repeats the create | ||
|
|
||
| >>> [CLI] bundle deploy | ||
| Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files... | ||
| Created postgres_projects.my_project | ||
| Warn: deploying resources.postgres_branches.dev: retrying after 504 Gateway Timeout from POST /api/2.0/postgres/projects/test-pg-proj/branches | ||
| Created postgres_branches.dev | ||
| Files: 0 uploaded, 0 deleted | ||
| Resources: 2 created, 0 changed, 0 deleted, 0 unchanged | ||
|
|
||
| >>> print_requests.py --get //postgres/projects/test-pg-proj/branches | ||
| POST /api/2.0/postgres/projects/test-pg-proj/branches | ||
| GET /api/2.0/postgres/projects/test-pg-proj/branches/dev | ||
| POST /api/2.0/postgres/projects/test-pg-proj/branches | ||
| GET /api/2.0/postgres/projects/test-pg-proj/branches/dev/operations/[UUID] |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,12 @@ | ||
| title "The create request times out but the branch was created: deploy uses the branch" | ||
| fault.py --after-handler "POST /api/2.0/postgres/projects/*/branches" 504 0 1 DEADLINE_EXCEEDED | ||
| trace $CLI bundle deploy | ||
| trace print_requests.py --get //postgres/projects/test-pg-proj/branches | jq -r '.method + " " + .path' | ||
| $CLI bundle destroy --auto-approve > /dev/null 2>&1 | ||
| rm -f "$OUT_REQUESTS" | ||
|
|
||
| title "The create request times out and the branch was not created: deploy repeats the create" | ||
| fault.py "POST /api/2.0/postgres/projects/*/branches" 504 0 1 DEADLINE_EXCEEDED | ||
| trace $CLI bundle deploy | ||
| trace print_requests.py --get //postgres/projects/test-pg-proj/branches | jq -r '.method + " " + .path' | ||
| $CLI bundle destroy --auto-approve > /dev/null 2>&1 |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,6 @@ | ||
| Cloud = false | ||
| RecordRequests = true | ||
| EnvMatrix.READPLAN = [] | ||
| Env.DATABRICKS_BUNDLE_ENGINE = "direct" | ||
| Env.DATABRICKS_BUNDLE_RETRY_INTERVAL_MS = "100" | ||
| SuppressLines.warn_retry = "" |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -2,10 +2,15 @@ package dresources | |
|
|
||
| import ( | ||
| "context" | ||
| "fmt" | ||
| "slices" | ||
| "time" | ||
|
|
||
| "github.com/databricks/cli/bundle/config/resources" | ||
| "github.com/databricks/cli/libs/log" | ||
| "github.com/databricks/cli/libs/retry" | ||
| "github.com/databricks/databricks-sdk-go" | ||
| "github.com/databricks/databricks-sdk-go/apierr" | ||
| "github.com/databricks/databricks-sdk-go/common/types/fieldmask" | ||
| sdktime "github.com/databricks/databricks-sdk-go/common/types/time" | ||
| "github.com/databricks/databricks-sdk-go/marshal" | ||
|
|
@@ -125,7 +130,7 @@ func (r *ResourcePostgresBranch) DoCreate(ctx context.Context, config *PostgresB | |
| ForceSendFields: nil, | ||
| }) | ||
| if err != nil { | ||
| return "", nil, err | ||
| return r.recoverFailedCreate(ctx, config, err) | ||
| } | ||
|
|
||
| // Wait for the branch to be ready (long-running operation) | ||
|
|
@@ -138,6 +143,61 @@ func (r *ResourcePostgresBranch) DoCreate(ctx context.Context, config *PostgresB | |
| return remote.Name, remote, nil | ||
| } | ||
|
|
||
| // branchReadyTimeout bounds the wait for a branch that recoverFailedCreate found in INIT. | ||
| const branchReadyTimeout = 10 * time.Minute | ||
|
|
||
| // recoverFailedCreate handles a create request that failed with a transient error. The backend | ||
| // answers a slow create with 504 DEADLINE_EXCEEDED ("may still be completing"), so the branch may | ||
| // exist anyway: use it once it is ready instead of failing, or, if it does not exist, let the | ||
| // caller repeat the create. | ||
| func (r *ResourcePostgresBranch) recoverFailedCreate(ctx context.Context, config *PostgresBranchState, createErr error) (string, *PostgresBranchRemote, error) { | ||
| if !retry.IsTransient(ctx, createErr) { | ||
| return "", nil, createErr | ||
| } | ||
| // With replace_existing, a branch found now may predate the request, and repeating the | ||
| // request is safe. | ||
| if config.ReplaceExisting != nil && *config.ReplaceExisting { | ||
| return "", nil, retrySafe(createErr) | ||
| } | ||
|
|
||
| name := config.Parent + "/branches/" + config.BranchId | ||
| deadline := time.Now().Add(branchReadyTimeout) | ||
| for { | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. minor: Up to 10 minutes of polling inside DoCreate, before state is saved and outside the max-wait cap
Concrete effects:
The existing success path also waits inline ( Agreed by 3 reviewer model(s). |
||
| branch, err := retry.With(ctx, retry.IsTransientRead, func() (*postgres.Branch, error) { | ||
| return r.client.Postgres.GetBranch(ctx, postgres.GetBranchRequest{Name: name}) | ||
| }) | ||
| if apierr.IsMissing(err) { | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. minor: In the "may still be completing" case, the repeated create can end in ALREADY_EXISTS The backend's 504 says the request "may still be completing", yet the branch is read right away (:166) with no pause. If the backend has not committed the branch yet, the read returns 404. The code then returns The PR lists this as a known gap. But its own argument (the first attempt got a 504, not a 409, so no branch existed beforehand) applies equally to a 409 that follows our own transient failure in the same
Without one of these, the fix only helps when the branch already exists when the 504 arrives, which is the case the Agreed by 3 reviewer model(s). |
||
| return "", nil, retrySafe(createErr) | ||
| } | ||
| if err != nil { | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. nit: A cancelled context during the recovery read is logged as a failed existence check and replaced by the create error If Agreed by 3 reviewer model(s). |
||
| log.Warnf(ctx, "checking whether %s was created: %s", name, err) | ||
| return "", nil, createErr | ||
| } | ||
| var state postgres.BranchStatusState | ||
| if branch.Status != nil { | ||
| state = branch.Status.CurrentState | ||
| } | ||
| if state == postgres.BranchStatusStateReady { | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. minor: A pre-existing READY branch can be adopted after any transient error, not only a handler timeout The PR argues that without In those cases a The window is narrow (a 5xx must coincide with an ID collision), but the outcome is a silent takeover. A cheap guard: record Agreed by 3 reviewer model(s). |
||
| log.Warnf(ctx, "the branch was created despite the failed create request: %s", createErr) | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. major: The new adoption warning is not suppressed, so the cloud tests the PR targets still fail when the branch is adopted When recovery adopts the branch (found The two tests the description names as affected ( In Suggested fix: either log the adoption below Warn, or add a second root Agreed by 3 reviewer model(s). |
||
| remote := makePostgresBranchRemote(branch) | ||
| return remote.Name, remote, nil | ||
| } | ||
| // Only a new branch still initializing is worth waiting for; any other state (e.g. DELETED, | ||
| // ARCHIVED) will not become READY. | ||
| if state != postgres.BranchStatusStateInit { | ||
| return "", nil, fmt.Errorf("%w; branch %s exists in state %q", createErr, name, state) | ||
| } | ||
| if time.Now().After(deadline) { | ||
| return "", nil, fmt.Errorf("%w; branch %s exists but did not become ready within %s", createErr, name, branchReadyTimeout) | ||
| } | ||
| select { | ||
| case <-ctx.Done(): | ||
| return "", nil, ctx.Err() | ||
| case <-time.After(5 * time.Second): | ||
| } | ||
| } | ||
| } | ||
|
|
||
| // expire_time, no_expiry and ttl are three sides of one oneof, and the API accepts them | ||
| // in update_mask only under the group name: masking the field itself is answered with | ||
| // "Unknown field path in update_mask". Probed against a real workspace on 2026-08-31. | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
major: Most of recoverFailedCreate is untested: INIT wait, non-INIT state, timeout, read failure, replace_existing
create_504exercises two outcomes only: the branch is foundREADYon the first read, and the branch is missing so the create is repeated. These paths ofrecoverFailedCreatehave no coverage:INITpoll loop (about lines 187-197), which the PR description presents as a main new behavior and which a "may still be completing" 504 is likely to hit;INIT/non-READYstate error (line 188) and the 10-minute timeout (line 191);replace_existing: true(lines 159-161), which repeats the create without reading and is the documented way to manage the implicitproductionbranch;The PR says the
INITpath can't be tested because the testserver always creates branches asREADY(libs/testserver/postgres.go:347-349). That rules out an acceptance test forINIT, but not a unit test. Other resources in this package testDoCreateagainstmocks.NewMockWorkspaceClient(t)(genie_space_test.go:60,feature_spec_test.go:44; the mock exposesGetMockPostgresAPI()), andjob_run_test.go:39-66drives a wait withtestserver.New(t)and a customserver.Handle. There is nopostgres_branch_test.go. A table-driven test could cover: GET returnsINITthenREADY; GET returns an unexpected state (the error carries the original create error); GET fails with a non-missing error (the original error is returned);replace_existing: true(no GET,RetrySafeError); a non-transient create error (no GET).Two literals block such a test: the poll interval
time.After(5 * time.Second)(:196) ignoresDATABRICKS_BUNDLE_RETRY_INTERVAL_MS, and the deadline is theconst branchReadyTimeout(:147). AnINITtest would sleep 5s and the timeout path can't be reached. Make them fields or package vars a test can shrink.Two of the gaps can also be closed with acceptance scenarios and no testserver changes, using the existing fault harness (
acceptance/bin/fault.pysupports--after-handlerand arbitrary method/path patterns):replace_existing: truewithfault.py --after-handler "POST /api/2.0/postgres/projects/*/branches" 504 0 1 DEADLINE_EXCEEDEDshould logPOST,POST, operationGET, and no branchGET.fault.py "GET /api/2.0/postgres/projects/*/branches/dev" 500 0 3after a 504 on the POST should show thechecking whether ... was createdwarning and the original 504 error, with no repeated POST.These tests don't duplicate
create_504: that covers the READY and missing paths, and these cover paths it can't reach.Agreed by 3 reviewer model(s).