Skip to content

Retry workspace requests after the SDK's inactivity timeout - #7521

Merged
denik merged 2 commits into
mainfrom
fix-import-file-inactivity-timeout
Nov 12, 2026
Merged

denik merged 2 commits into
mainfrom
fix-import-file-inactivity-timeout

Conversation

@denik

@denik denik commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Changes

  • libs/retry.IsTransient now also returns true for the SDK's client-side inactivity timeout (*url.Error with "request timed out after ... of inactivity"). Callers already use it only for calls that are safe to repeat: overwrite writes and deletes through retry.Filer, reads through IsTransientRead, and the direct engine's update/delete/wait calls. Non-overwrite writes, such as the deploy lock create, are still not retried.
  • Retry warnings for non-API errors now read retrying after <error> instead of a bare retrying, so the warning says what happened and acceptance tests suppress it like the other retry warnings (SuppressLines.warn_retry).
  • acceptance/bundle/upload/timeout now records the two retries and the three upload attempts.
  • Changelog entry in .nextchanges/bundles/7521.md.

Why

When the workspace filesystem API sends nothing for HTTPTimeout (90s for bundles), the SDK fails the request and never retries it. Its retry check only sees API errors and a short list of transport errors. libs/retry (#7129) fills the gaps for 5xx/408 but not for this timeout. This is the most common cloud-test failure: about 10k failures in 120 days on import-file alone, mostly on burst days that hit AWS, Azure and GCP together. The same timeout also shows up on workspace/get-status, workspace/import and GET workspace-files.

This is not a CLI regression. The stalls are server-side, and a test-level rerun sometimes succeeds (about 1 in 3 affected tests end up flaky rather than failed). A timeout carries the same uncertainty as a 504 (the server may or may not have applied the request), and we already retry 504s at these call sites.

Judgment call: a stalled attempt takes up to HTTPTimeout (90s) before the retry. With MaxRetries = 2 and the 15s pause, a deploy that hits a persistent stall can now take about 5 minutes longer before it fails. That matches the intent in bundle/config/workspace.go ("do utmost to complete" a deploy).

Known edge cases: these call sites are already retried on 504, and a timeout now reaches them too. A UC rename that is applied but times out is re-sent with the old name and fails with a 404 (it fails safely, with no data loss). A lock release that is applied but times out could, in a narrow window, delete a lock that another deployer took in the meantime.

Overlap with #7518: #7518 widens the acceptance SuppressLines.warn_retry filter to also match the bare retrying warning. This PR replaces that bare form with retrying after <error> at the source, so the current filter already matches it. The two PRs don't touch the same files, and either merge order works; #7518's wider filter stays as a safety net.

Tests

  • Unit tests in libs/retry for the timeout classification, including the wrapped and cancelled cases.
  • acceptance/bundle/upload/timeout shows the retries.
  • Full local acceptance suite passes. The only exceptions are tests that record the user agent, which pick up an extra agent-environment token on the machine I ran them on; their only diff is the user-agent string.

Dashboard:

error snippet
error/179 Error: Post "[DATABRICKS_URL]/api/2.0/workspace-files/import-file/[PATH]": request timed out after 1m30s of inactivity

This pull request and its description were written by Isaac.

The SDK gives up on a request when the server sends nothing for HTTPTimeout
("request timed out after 1m30s of inactivity") and never retries it.
libs/retry already retries 5xx and 408 for overwrite writes, reads and
deletes; treat this timeout the same way. Print non-API retry reasons as
"retrying after <err>" so the warning says what happened and acceptance
tests suppress it like the other retry warnings.

Co-authored-by: Isaac <no-reply@databricks.com>
@denik denik added the ai-auto PRs made by AI loops & factories, label Nov 11, 2026
@github-actions github-actions Bot added the DABs DABs related issues label Nov 11, 2026
Co-authored-by: Isaac <no-reply@databricks.com>
@github-actions

Copy link
Copy Markdown
Contributor

Commit: e79c1d9dc | Workflow run

Regression test report

Selected 1 changed tests (limit=20, 0 not selected)

2 tests: regression=2 unreleased=0 coverage=0 failing=0
test branch main (611f4e8) latest (v1.26.0)
TestAccept/bundle/upload/timeout/DMS= ✅ ❌ ❌
TestAccept/bundle/upload/timeout/DMS=true ✅ ❌ ❌
TestAccept/bundle/upload/timeout/DMS=: branch ✅ | main (611f4e8) ❌ | latest (v1.26.0) ❌

main (611f4e8):

prepare_server.go:269: request canceled: context canceled
    acceptance_test.go:1345: Diff:
        --- bundle/upload/timeout/output.txt
        +++ /tmp/TestAcceptbundleuploadtimeoutDMS=1485345164/002/output.txt
        @@ -1,8 +1,4 @@
         Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files...
        -Warn: retrying after Post "[DATABRICKS_URL]/api/2.0/workspace-files/import-file/Workspace%2FUsers%2F[USERNAME]%2F.bundle%2Ftest-bundle%2Fdefault%2Ffiles%2Ffile_to_upload.txt?overwrite=true": request timed out after 5s of inactivity
        -Warn: retrying after Post "[DATABRICKS_URL]/api/2.0/workspace-files/import-file/Workspace%2FUsers%2F[USERNAME]%2F.bundle%2Ftest-bundle%2Fdefault%2Ffiles%2Ffile_to_upload.txt?overwrite=true": request timed out after 5s of inactivity
         Error: Post "[DATABRICKS_URL]/api/2.0/workspace-files/import-file/Workspace%2FUsers%2F[USERNAME]%2F.bundle%2Ftest-bundle%2Fdefault%2Ffiles%2Ffile_to_upload.txt?overwrite=true": request timed out after 5s of inactivity
         
         /api/2.0/workspace-files/import-file/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files/file_to_upload.txt
        -/api/2.0/workspace-files/import-file/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files/file_to_upload.txt
        -/api/2.0/workspace-files/import-file/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files/file_to_upload.txt

latest (v1.26.0): same output as main (611f4e8)

TestAccept/bundle/upload/timeout/DMS=true: branch ✅ | main (611f4e8) ❌ | latest (v1.26.0) ❌

main (611f4e8):

prepare_server.go:269: request canceled: context canceled
    acceptance_test.go:1345: Diff:
        --- bundle/upload/timeout/output.txt
        +++ /tmp/TestAcceptbundleuploadtimeoutDMS=true2127832981/002/output.txt
        @@ -1,8 +1,4 @@
         Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files...
        -Warn: retrying after Post "[DATABRICKS_URL]/api/2.0/workspace-files/import-file/Workspace%2FUsers%2F[USERNAME]%2F.bundle%2Ftest-bundle%2Fdefault%2Ffiles%2Ffile_to_upload.txt?overwrite=true": request timed out after 5s of inactivity
        -Warn: retrying after Post "[DATABRICKS_URL]/api/2.0/workspace-files/import-file/Workspace%2FUsers%2F[USERNAME]%2F.bundle%2Ftest-bundle%2Fdefault%2Ffiles%2Ffile_to_upload.txt?overwrite=true": request timed out after 5s of inactivity
         Error: Post "[DATABRICKS_URL]/api/2.0/workspace-files/import-file/Workspace%2FUsers%2F[USERNAME]%2F.bundle%2Ftest-bundle%2Fdefault%2Ffiles%2Ffile_to_upload.txt?overwrite=true": request timed out after 5s of inactivity
         
         /api/2.0/workspace-files/import-file/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files/file_to_upload.txt
        -/api/2.0/workspace-files/import-file/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files/file_to_upload.txt
        -/api/2.0/workspace-files/import-file/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files/file_to_upload.txt
        
FAIL

latest (v1.26.0): same output as main (611f4e8)

@eng-dev-ecosystem-bot

Copy link
Copy Markdown
Collaborator

✅ Integration test report

Commit: e79c1d9

Run: 43119665016

Env 💚​RECOVERED ✅​pass 🙈​skip Time
💚​ aws linux-2core-8gb 1 270 16 7:04
💚​ aws-windows-latest-4core-16gb 1 272 14 6:09
💚​ azure linux-2core-8gb 1 269 16 6:14
💚​ azure-windows-latest-4core-16gb 1 271 14 4:11
💚​ gcp linux-2core-8gb 1 270 16 9:37
💚​ gcp-windows-latest-4core-16gb 1 272 14 5:17
Test Name aws linux-2core-8gb aws-windows-latest-4core-16gb azure linux-2core-8gb azure-windows-latest-4core-16gb gcp linux-2core-8gb gcp-windows-latest-4core-16gb
💚​ TestAccept 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
Top 8 slowest tests (at least 2 minutes):
duration env testname
4:04 gcp-windows-latest-4core-16gb TestAccept
3:57 gcp linux-2core-8gb TestAccept
3:56 azure linux-2core-8gb TestAccept
3:54 azure-windows-latest-4core-16gb TestAccept
3:52 aws linux-2core-8gb TestAccept
3:46 aws-windows-latest-4core-16gb TestAccept
2:35 gcp linux-2core-8gb TestTelemetryEndpoint
2:14 aws-windows-latest-4core-16gb TestTelemetryEndpoint

@denik
denik marked this pull request as ready for review November 11, 2026 23:50
@denik
denik requested review from a team as code owners November 11, 2026 23:50
@denik
denik requested a review from anton-107 November 11, 2026 23:50
@denik
denik enabled auto-merge November 11, 2026 23:50
@andrewnester
andrewnester self-requested a review November 12, 2026 08:59
Comment thread libs/retry/retry.go
return false
}
msg := urlErr.Err.Error()
return strings.HasPrefix(msg, "request timed out after ") && strings.HasSuffix(msg, " of inactivity")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This matches the SDK's message because httpclient/api_client.go builds it with fmt.Errorf and exports no sentinel; can the comment say so, or should the SDK export one?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed: the comment should say so. The SDK (v0.197.0) builds this error with fmt.Errorf("request timed out after %s of inactivity", ...) in httpclient/api_client.go and exports no sentinel or type, so the CLI has to match the message. #7541 adds that note to the comment. An exported sentinel in the SDK would be the cleaner fix; I've left that for the SDK side.

Written by the cli-tests-fix AI agent (harness_auto).

@denik
denik added this pull request to the merge queue Nov 12, 2026
Merged via the queue into main with commit 4450cc2 Nov 12, 2026
28 checks passed
@denik
denik deleted the fix-import-file-inactivity-timeout branch November 12, 2026 10:10
tedjt pushed a commit to tedjt/cli that referenced this pull request Nov 12, 2026
…ks#7541)

## Changes
Document in `libs/retry/retry.go` why `isInactivityTimeout` matches the
SDK's error message: the SDK builds "request timed out after ... of
inactivity" with `fmt.Errorf` in `httpclient/api_client.go` and exports
no sentinel error or type for it.

## Why
Follow-up to [databricks#7521](databricks#7521),
addressing review feedback that merged before it was handled. An
exported sentinel in the SDK would be the better long-term fix; until
one exists, the message match follows the SDK's exact format string
(prefix and suffix).

## Tests
Comment-only change; `go test ./libs/retry` passes.

This pull request and its description were written by Isaac.

Co-authored-by: Isaac <no-reply@databricks.com>
idelafuente-oc pushed a commit to idelafuente-oc/cli that referenced this pull request Nov 14, 2026
## Changes

`databricks fs cp/ls/cat/rm` now retry `Stat`, `Read`, `ReadDir` and
`Delete` against DBFS and the Files API on transient errors, using the
policy from `libs/retry` (5xx/408 the SDK did not retry, the SDK's
"request timed out after ... of inactivity", `unexpected EOF`,
`REFUSED_STREAM`; 2 retries, 15s apart).

databricks#7521 added the inactivity-timeout retry, but only bundle code wraps its
filers with `retry.Filer`; `cmd/fs` used the raw DBFS/Files clients, so
a single stalled `GET /api/2.0/dbfs/get-status` failed the whole copy.

Added `retry.FilerExceptWrites`, a variant of `retry.Filer` that passes
`Write` through, because `fs cp` writes are not safe to retry the same
way:
- without `--overwrite`, a retried create may hit the file its own first
attempt created, and `fs cp` would then report it as "skipped; already
exists";
- with `--overwrite`, `retry.Filer` buffers a non-seekable source (e.g.
a DBFS/Volumes read stream) in memory to replay it, and `fs cp` can copy
multi-GB files.

Question for reviewers: error/1213 below fails in `DbfsClient.Write` at
`POST /api/2.0/dbfs/create`, before any data is read. Retrying just that
call in overwrite mode would be safe and cheap; in non-overwrite mode a
retry that gets RESOURCE_ALREADY_EXISTS could return the original
timeout instead of "already exists". Worth doing in a follow-up?

## Why

Flaky `integration/cmd/fs` tests on cloud.

## Dashboard

| error | snippet |
|---|---|
|
[error/2939](http://go/cli/dashboard/error/2939?main=1&pr=1&end=2026-11-12T13:45:00Z&last=7d)
| `Get "[DATABRICKS_URL]/api/2.0/dbfs/get-status": request timed out
after 1m0s of inactivity` |

Tests:
[integration/cmd/fs.TestFsCpDirToDirFileNotOverwritten/dbfs_to_uc-volumes](http://go/cli/dashboard/test/integration/cmd/fs.TestFsCpDirToDirFileNotOverwritten/dbfs_to_uc-volumes?main=1&pr=1&end=2026-11-12T13:45:00Z&last=7d),
[integration/cmd/fs.TestFsCpFileToDir/dbfs_to_local](http://go/cli/dashboard/test/integration/cmd/fs.TestFsCpFileToDir/dbfs_to_local?main=1&pr=1&end=2026-11-12T13:45:00Z&last=7d)

Not covered here (same family, different paths):
-
[error/1213](http://go/cli/dashboard/error/1213?main=1&pr=1&end=2026-11-12T13:45:00Z&last=7d)
`dbfs open: Post "[DATABRICKS_URL]/api/2.0/dbfs/create": request timed
out after 1m0s of inactivity`: a non-overwrite write, see the question
above.
-
[error/3399](http://go/cli/dashboard/error/3399?main=1&pr=1&end=2026-11-12T13:45:00Z&last=7d)
`Put "[DATABRICKS_URL]/api/2.0/fs/files/[PATH]": read tcp ...`: a
test-setup write without overwrite, not a CLI call.
-
[error/2921](http://go/cli/dashboard/error/2921?main=1&pr=1&end=2026-11-12T13:45:00Z&last=7d)
`Get "[DATABRICKS_URL]/api/2.0/workspace/get-status": unexpected EOF`:
the generated `workspace get-status` command, which only goes through
the SDK.

## Tests

Unit tests for `retry.FilerExceptWrites`; new
`acceptance/cmd/fs/retry-504` injects a 504 on dbfs list, get-status and
delete and shows each retried; `cmd/fs` unit tests and local
`TestAccept/cmd/fs` pass.

This pull request and its description were written by Isaac.

---------

Co-authored-by: Isaac <no-reply@databricks.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ai-auto PRs made by AI loops & factories, DABs DABs related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants