Skip to content

Retry fs command reads on transient errors - #7556

Merged
denik merged 3 commits into
mainfrom
fs-retry-reads
Nov 12, 2026
Merged

denik merged 3 commits into
mainfrom
fs-retry-reads

Conversation

@denik

@denik denik commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Changes

databricks fs cp/ls/cat/rm now retry Stat, Read, ReadDir and Delete against DBFS and the Files API on transient errors, using the policy from libs/retry (5xx/408 the SDK did not retry, the SDK's "request timed out after ... of inactivity", unexpected EOF, REFUSED_STREAM; 2 retries, 15s apart).

#7521 added the inactivity-timeout retry, but only bundle code wraps its filers with retry.Filer; cmd/fs used the raw DBFS/Files clients, so a single stalled GET /api/2.0/dbfs/get-status failed the whole copy.

Added retry.FilerExceptWrites, a variant of retry.Filer that passes Write through, because fs cp writes are not safe to retry the same way:

  • without --overwrite, a retried create may hit the file its own first attempt created, and fs cp would then report it as "skipped; already exists";
  • with --overwrite, retry.Filer buffers a non-seekable source (e.g. a DBFS/Volumes read stream) in memory to replay it, and fs cp can copy multi-GB files.

Question for reviewers: error/1213 below fails in DbfsClient.Write at POST /api/2.0/dbfs/create, before any data is read. Retrying just that call in overwrite mode would be safe and cheap; in non-overwrite mode a retry that gets RESOURCE_ALREADY_EXISTS could return the original timeout instead of "already exists". Worth doing in a follow-up?

Why

Flaky integration/cmd/fs tests on cloud.

Dashboard

error snippet
error/2939 Get "[DATABRICKS_URL]/api/2.0/dbfs/get-status": request timed out after 1m0s of inactivity

Tests: integration/cmd/fs.TestFsCpDirToDirFileNotOverwritten/dbfs_to_uc-volumes, integration/cmd/fs.TestFsCpFileToDir/dbfs_to_local

Not covered here (same family, different paths):

  • error/1213 dbfs open: Post "[DATABRICKS_URL]/api/2.0/dbfs/create": request timed out after 1m0s of inactivity: a non-overwrite write, see the question above.
  • error/3399 Put "[DATABRICKS_URL]/api/2.0/fs/files/[PATH]": read tcp ...: a test-setup write without overwrite, not a CLI call.
  • error/2921 Get "[DATABRICKS_URL]/api/2.0/workspace/get-status": unexpected EOF: the generated workspace get-status command, which only goes through the SDK.

Tests

Unit tests for retry.FilerExceptWrites; new acceptance/cmd/fs/retry-504 injects a 504 on dbfs list, get-status and delete and shows each retried; cmd/fs unit tests and local TestAccept/cmd/fs pass.

This pull request and its description were written by Isaac.

databricks fs cp/ls/cat/rm now retry Stat, Read and ReadDir against DBFS and
the Files API on transient errors, including the SDK's inactivity timeout,
using the same retry policy as bundle file operations (libs/retry).

Writes and deletes are not retried.

Co-authored-by: Isaac <no-reply@databricks.com>
@denik denik added the ai-auto PRs made by AI loops & factories, label Nov 12, 2026
Co-authored-by: Isaac <no-reply@databricks.com>
@denik
denik requested a review from andrewnester November 12, 2026 13:50
@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Collaborator

✅ Integration test report

Commit: f153cb2

Run: 43243287563

Env 🔄​flaky 💚​RECOVERED ✅​pass 🙈​skip Time
💚​ aws linux-2core-8gb 1 270 16 9:20
💚​ aws-windows-latest-4core-16gb 1 272 14 6:19
💚​ azure linux-2core-8gb 1 269 16 12:02
💚​ azure-windows-latest-4core-16gb 1 271 14 8:13
💚​ gcp linux-2core-8gb 1 270 16 11:23
🔄​ gcp-windows-latest-4core-16gb 2 1 270 14 9:17
Test Name aws linux-2core-8gb aws-windows-latest-4core-16gb azure linux-2core-8gb azure-windows-latest-4core-16gb gcp linux-2core-8gb gcp-windows-latest-4core-16gb
💚​ TestAccept 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
🔄​ TestSyncIncrementalFileSync ✅​p ✅​p ✅​p ✅​p ✅​p 🔄​f
🔄​ TestSyncIncrementalSyncPythonNotebookDelete ✅​p ✅​p ✅​p ✅​p ✅​p 🔄​f
Top 14 slowest tests (at least 2 minutes):
duration env testname
4:02 aws linux-2core-8gb TestAccept
3:54 gcp linux-2core-8gb TestAccept
3:54 azure linux-2core-8gb TestFilerWorkspaceFilesExtensionsReadDir
3:48 azure linux-2core-8gb TestAccept
3:47 aws-windows-latest-4core-16gb TestAccept
3:39 azure-windows-latest-4core-16gb TestAccept
3:30 gcp-windows-latest-4core-16gb TestAccept
3:15 gcp linux-2core-8gb TestFilerWorkspaceFilesExtensionsReadDir
3:12 azure linux-2core-8gb TestImportDirWithOverwriteFlag
2:59 gcp-windows-latest-4core-16gb TestFilerRecursiveDelete/workspace_files_extensions
2:52 azure-windows-latest-4core-16gb TestImportDirDoesNotOverwrite
2:44 azure-windows-latest-4core-16gb TestFilerRecursiveDelete/workspace_files_extensions
2:26 aws linux-2core-8gb TestFilerWorkspaceFilesExtensionsReadDir
2:23 gcp-windows-latest-4core-16gb TestFilerRecursiveDelete/workspace_files

@denik
denik marked this pull request as ready for review November 12, 2026 14:21
@denik
denik requested review from a team as code owners November 12, 2026 14:21
@denik
denik requested a review from rauchy November 12, 2026 14:21
Comment thread libs/retry/filer.go Resolved
Comment thread cmd/fs/helpers.go Resolved
Rename retry.ReadFiler to retry.FilerExceptWrites: it keeps Filer's
retry-safe Delete and skips only Write. Add acceptance/cmd/fs/retry-504,
which injects a 504 on dbfs list, get-status and delete.

Co-authored-by: Isaac <no-reply@databricks.com>
@denik
denik requested a review from andrewnester November 12, 2026 15:00
@github-actions

Copy link
Copy Markdown
Contributor

Commit: f153cb29c | Workflow run

Regression test report

Selected 1 changed tests (limit=20, 0 not selected)

1 tests: regression=1 unreleased=0 coverage=0 failing=0
test branch main (2848f68) latest (v1.26.0)
TestAccept/cmd/fs/retry-504 ✅ ❌ ❌
TestAccept/cmd/fs/retry-504: branch ✅ | main (2848f68) ❌ | latest (v1.26.0) ❌

main (2848f68):

acceptance_test.go:1373: 
        	Error Trace:	/home/runner/work/cli/cli/libs/testdiff/testdiff.go:22
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1373
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1114
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:660
        	Error:      	Not equal: 
        	            	expected: "\n=== A 504 on list is retried\n\n>>> [CLI] fs ls dbfs:/tmp/fs-retry-504-[UNIQUE_NAME]\nWarn: retrying after 504 Gateway Timeout from GET /api/2.0/dbfs/list\nhello.txt\n\n=== A 504 on get-status is retried\n\n>>> [CLI] fs cat dbfs:/tmp/fs-retry-504-[UNIQUE_NAME]/hello.txt\nWarn: retrying after 504 Gateway Timeout from GET /api/2.0/dbfs/get-status: dbfs open: Fault injected by test.\nhello\n\n=== A 504 on delete is retried\n\n>>> [CLI] fs rm dbfs:/tmp/fs-retry-504-[UNIQUE_NAME]/hello.txt\nWarn: retrying after 504 Gateway Timeout from POST /api/2.0/dbfs/delete\n\n>>> [CLI] fs ls dbfs:/tmp/fs-retry-504-[UNIQUE_NAME]\n"
        	            	actual  : "\n=== A 504 on list is retried\n\n>>> [CLI] fs ls dbfs:/tmp/fs-retry-504-[UNIQUE_NAME]\nError: Fault injected by test.\n\nExit code: 1\n"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -4,17 +4,5 @@
        	            	 >>> [CLI] fs ls dbfs:/tmp/fs-retry-504-[UNIQUE_NAME]
        	            	-Warn: retrying after 504 Gateway Timeout from GET /api/2.0/dbfs/list
        	            	-hello.txt
        	            	+Error: Fault injected by test.
        	            	 
        	            	-=== A 504 on get-status is retried
        	            	+Exit code: 1
        	            	 
        	            	->>> [CLI] fs cat dbfs:/tmp/fs-retry-504-[UNIQUE_NAME]/hello.txt
        	            	-Warn: retrying after 504 Gateway Timeout from GET /api/2.0/dbfs/get-status: dbfs open: Fault injected by test.
        	            	-hello
        	            	-
        	            	-=== A 504 on delete is retried
        	            	-
        	            	->>> [CLI] fs rm dbfs:/tmp/fs-retry-504-[UNIQUE_NAME]/hello.txt
        	            	-Warn: retrying after 504 Gateway Timeout from POST /api/2.0/dbfs/delete
        	            	-
        	            	->>> [CLI] fs ls dbfs:/tmp/fs-retry-504-[UNIQUE_NAME]
        	            	-
        	Test:       	TestAccept/cmd/fs/retry-504
        	Messages:   	cmd/fs/retry-504/output.txt vs /tmp/TestAcceptcmdfsretry-5042533894287/002/output.txt
FAIL

latest (v1.26.0): same output as main (2848f68)

@denik
denik enabled auto-merge November 12, 2026 17:37
@denik
denik added this pull request to the merge queue Nov 12, 2026
Merged via the queue into main with commit 45285b9 Nov 12, 2026
28 checks passed
@denik
denik deleted the fs-retry-reads branch November 12, 2026 18:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ai-auto PRs made by AI loops & factories,

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants