Repository navigation
Retry workspace requests after the SDK's inactivity timeout - #7521
Conversation
The SDK gives up on a request when the server sends nothing for HTTPTimeout
("request timed out after 1m30s of inactivity") and never retries it.
libs/retry already retries 5xx and 408 for overwrite writes, reads and
deletes; treat this timeout the same way. Print non-API retry reasons as
"retrying after <err>" so the warning says what happened and acceptance
tests suppress it like the other retry warnings.
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
|
Commit: Regression test reportSelected 1 changed tests (limit=20, 0 not selected) 2 tests: regression=2 unreleased=0 coverage=0 failing=0
TestAccept/bundle/upload/timeout/DMS=: branch ✅ | main (611f4e8) ❌ | latest (v1.26.0) ❌main (611f4e8): latest (v1.26.0): same output as main (611f4e8) TestAccept/bundle/upload/timeout/DMS=true: branch ✅ | main (611f4e8) ❌ | latest (v1.26.0) ❌main (611f4e8): latest (v1.26.0): same output as main (611f4e8) |
✅ Integration test reportCommit: e79c1d9
Top 8 slowest tests (at least 2 minutes):
|
| return false | ||
| } | ||
| msg := urlErr.Err.Error() | ||
| return strings.HasPrefix(msg, "request timed out after ") && strings.HasSuffix(msg, " of inactivity") |
There was a problem hiding this comment.
This matches the SDK's message because httpclient/api_client.go builds it with fmt.Errorf and exports no sentinel; can the comment say so, or should the SDK export one?
There was a problem hiding this comment.
Agreed: the comment should say so. The SDK (v0.197.0) builds this error with fmt.Errorf("request timed out after %s of inactivity", ...) in httpclient/api_client.go and exports no sentinel or type, so the CLI has to match the message. #7541 adds that note to the comment. An exported sentinel in the SDK would be the cleaner fix; I've left that for the SDK side.
Written by the cli-tests-fix AI agent (harness_auto).
…ks#7541) ## Changes Document in `libs/retry/retry.go` why `isInactivityTimeout` matches the SDK's error message: the SDK builds "request timed out after ... of inactivity" with `fmt.Errorf` in `httpclient/api_client.go` and exports no sentinel error or type for it. ## Why Follow-up to [databricks#7521](databricks#7521), addressing review feedback that merged before it was handled. An exported sentinel in the SDK would be the better long-term fix; until one exists, the message match follows the SDK's exact format string (prefix and suffix). ## Tests Comment-only change; `go test ./libs/retry` passes. This pull request and its description were written by Isaac. Co-authored-by: Isaac <no-reply@databricks.com>
## Changes `databricks fs cp/ls/cat/rm` now retry `Stat`, `Read`, `ReadDir` and `Delete` against DBFS and the Files API on transient errors, using the policy from `libs/retry` (5xx/408 the SDK did not retry, the SDK's "request timed out after ... of inactivity", `unexpected EOF`, `REFUSED_STREAM`; 2 retries, 15s apart). databricks#7521 added the inactivity-timeout retry, but only bundle code wraps its filers with `retry.Filer`; `cmd/fs` used the raw DBFS/Files clients, so a single stalled `GET /api/2.0/dbfs/get-status` failed the whole copy. Added `retry.FilerExceptWrites`, a variant of `retry.Filer` that passes `Write` through, because `fs cp` writes are not safe to retry the same way: - without `--overwrite`, a retried create may hit the file its own first attempt created, and `fs cp` would then report it as "skipped; already exists"; - with `--overwrite`, `retry.Filer` buffers a non-seekable source (e.g. a DBFS/Volumes read stream) in memory to replay it, and `fs cp` can copy multi-GB files. Question for reviewers: error/1213 below fails in `DbfsClient.Write` at `POST /api/2.0/dbfs/create`, before any data is read. Retrying just that call in overwrite mode would be safe and cheap; in non-overwrite mode a retry that gets RESOURCE_ALREADY_EXISTS could return the original timeout instead of "already exists". Worth doing in a follow-up? ## Why Flaky `integration/cmd/fs` tests on cloud. ## Dashboard | error | snippet | |---|---| | [error/2939](http://go/cli/dashboard/error/2939?main=1&pr=1&end=2026-11-12T13:45:00Z&last=7d) | `Get "[DATABRICKS_URL]/api/2.0/dbfs/get-status": request timed out after 1m0s of inactivity` | Tests: [integration/cmd/fs.TestFsCpDirToDirFileNotOverwritten/dbfs_to_uc-volumes](http://go/cli/dashboard/test/integration/cmd/fs.TestFsCpDirToDirFileNotOverwritten/dbfs_to_uc-volumes?main=1&pr=1&end=2026-11-12T13:45:00Z&last=7d), [integration/cmd/fs.TestFsCpFileToDir/dbfs_to_local](http://go/cli/dashboard/test/integration/cmd/fs.TestFsCpFileToDir/dbfs_to_local?main=1&pr=1&end=2026-11-12T13:45:00Z&last=7d) Not covered here (same family, different paths): - [error/1213](http://go/cli/dashboard/error/1213?main=1&pr=1&end=2026-11-12T13:45:00Z&last=7d) `dbfs open: Post "[DATABRICKS_URL]/api/2.0/dbfs/create": request timed out after 1m0s of inactivity`: a non-overwrite write, see the question above. - [error/3399](http://go/cli/dashboard/error/3399?main=1&pr=1&end=2026-11-12T13:45:00Z&last=7d) `Put "[DATABRICKS_URL]/api/2.0/fs/files/[PATH]": read tcp ...`: a test-setup write without overwrite, not a CLI call. - [error/2921](http://go/cli/dashboard/error/2921?main=1&pr=1&end=2026-11-12T13:45:00Z&last=7d) `Get "[DATABRICKS_URL]/api/2.0/workspace/get-status": unexpected EOF`: the generated `workspace get-status` command, which only goes through the SDK. ## Tests Unit tests for `retry.FilerExceptWrites`; new `acceptance/cmd/fs/retry-504` injects a 504 on dbfs list, get-status and delete and shows each retried; `cmd/fs` unit tests and local `TestAccept/cmd/fs` pass. This pull request and its description were written by Isaac. --------- Co-authored-by: Isaac <no-reply@databricks.com>
Changes
libs/retry.IsTransientnow also returns true for the SDK's client-side inactivity timeout (*url.Errorwith "request timed out after ... of inactivity"). Callers already use it only for calls that are safe to repeat: overwrite writes and deletes throughretry.Filer, reads throughIsTransientRead, and the direct engine's update/delete/wait calls. Non-overwrite writes, such as the deploy lock create, are still not retried.retrying after <error>instead of a bareretrying, so the warning says what happened and acceptance tests suppress it like the other retry warnings (SuppressLines.warn_retry).acceptance/bundle/upload/timeoutnow records the two retries and the three upload attempts..nextchanges/bundles/7521.md.Why
When the workspace filesystem API sends nothing for
HTTPTimeout(90s for bundles), the SDK fails the request and never retries it. Its retry check only sees API errors and a short list of transport errors.libs/retry(#7129) fills the gaps for 5xx/408 but not for this timeout. This is the most common cloud-test failure: about 10k failures in 120 days onimport-filealone, mostly on burst days that hit AWS, Azure and GCP together. The same timeout also shows up onworkspace/get-status,workspace/importand GETworkspace-files.This is not a CLI regression. The stalls are server-side, and a test-level rerun sometimes succeeds (about 1 in 3 affected tests end up flaky rather than failed). A timeout carries the same uncertainty as a 504 (the server may or may not have applied the request), and we already retry 504s at these call sites.
Judgment call: a stalled attempt takes up to
HTTPTimeout(90s) before the retry. WithMaxRetries = 2and the 15s pause, a deploy that hits a persistent stall can now take about 5 minutes longer before it fails. That matches the intent inbundle/config/workspace.go("do utmost to complete" a deploy).Known edge cases: these call sites are already retried on 504, and a timeout now reaches them too. A UC rename that is applied but times out is re-sent with the old name and fails with a 404 (it fails safely, with no data loss). A lock release that is applied but times out could, in a narrow window, delete a lock that another deployer took in the meantime.
Overlap with #7518: #7518 widens the acceptance
SuppressLines.warn_retryfilter to also match the bareretryingwarning. This PR replaces that bare form withretrying after <error>at the source, so the current filter already matches it. The two PRs don't touch the same files, and either merge order works; #7518's wider filter stays as a safety net.Tests
libs/retryfor the timeout classification, including the wrapped and cancelled cases.acceptance/bundle/upload/timeoutshows the retries.Dashboard:
Error: Post "[DATABRICKS_URL]/api/2.0/workspace-files/import-file/[PATH]": request timed out after 1m30s of inactivityThis pull request and its description were written by Isaac.