Repository navigation
filer: retry throttled Files API calls on UC Volumes - #7559
Merged
Merged
Conversation
Since the filer moved to the sdk-go files/v2 client, Files API calls are
no longer retried: files/v2 retries nothing unless the caller passes a
retrier. The backend reports throttled cloud storage ("AWS S3 is
throttling requests; try again later.") as 429 RESOURCE_EXHAUSTED and
expects the caller to retry.
Retry 429 and 503 on every files/v2 call the filer makes. The single-shot
upload cannot take a retrier, so the filer retries it itself, rewinding a
seekable reader to its starting offset before each attempt.
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
…reuse Use apiretry.NewRetrier with a retry cap, so Retry-After hints and transient network errors are honored, and retry the same statuses files/v2 retries for its own control-plane calls (408, 429, 502, 503, 504). Wrap each upload attempt's body so a late read by net/http cannot race with rewinding the reader for the next attempt. Co-authored-by: Isaac <no-reply@databricks.com>
Collaborator
✅ Integration test reportCommit: 59079aa
Top 10 slowest tests (at least 2 minutes):
|
andrewnester
approved these changes
Nov 12, 2026
andrewnester
left a comment
Contributor
There was a problem hiding this comment.
.nextchanges/cli/7559.md: the entry says "HTTP 429 or 503", but the set is 408/429/502/503/504 plus transient network errors.
Co-authored-by: Isaac <no-reply@databricks.com>
Contributor
Author
|
Thanks @andrewnester, agreed. AI agent (harness_auto, opus5.5, isaac_claude) |
andrewnester
approved these changes
Nov 12, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
Retry Files API calls made by
libs/filer.FilesClient(UC Volumes:fs cp/ls/rm/cat/mkdir, bundle uploads to Volumes) on HTTP 408, 429, 502, 503 and 504 and on transient network errors. A call gets up to 10 retries with 1s..10s jittered exponential backoff, and aRetry-Afterhint is honored.files/v2call (metadata, download, delete, list, mkdir) getscall.WithRetrier(...).UploadFilestreams its request body, andfiles/v2rejects a retrier on it. The filer retries it itself withops.Execute: a seekable reader (local files,strings.Reader,bytes.Reader) is rewound to the offset it had whenWritewas called before each attempt. Each attempt's body is wrapped, so a late read bynet/httpfrom an earlier attempt cannot race with the rewind. A non-seekable reader, including a pipe (an*os.FilewhoseSeekfails), still gets one attempt.DATABRICKS_EXPERIMENTAL_MULTIPART_UPLOAD) already retries its control-plane calls and is unchanged.Why
This is a regression from #6018 (2026-07-29). That PR moved the filer from
databricks-sdk-go'sclient.Doto thesdk-go/files/v2client. The old client retried 429 and 503 automatically.files/v2retries nothing unless the caller passes a retrier (options/call: "If no retrier is provided, the call is not retried"), and the filer never passed one.When cloud storage throttles, the Files API returns 429
RESOURCE_EXHAUSTEDwith the message "AWS S3 is throttling requests; try again later." The caller is expected to retry, so this is not a backend bug. Without retries, a single throttled request fails the whole command.Decisions:
core/apiretry.NewRetrier(the onefiles/v2uses for its control-plane calls) with a retry cap, and the status set is the same as that path's. The filer's single-shot calls now behave like the multipart engine's calls.UploadFileopting out) would be the more robust long-term fix.Tests
libs/filer/files_client_test.gocover: throttled metadata, upload, download and delete calls are retried; a retried upload resends the body from the reader's starting offset; a non-seekable upload is not retried; retries stop after the cap. Without the fix, the retry tests fail.TestFilerRecursiveDelete,TestFsCpDirToDirFileNotOverwritten,TestFsCpFileToDirWithOverwriteFlag,TestFsCpFileToFileWithOverwriteFlagandTestFsCpSourceIsDirectoryButTargetIsFile, all variants. All pass. Throttling did not reproduce during the run.go test ./acceptance -run 'TestAccept/cmd/fs|TestAccept/bundle/libraries'passes.Dashboard
Error: databricks-api error: AWS S3 is throttling requests; try again later.Flaky tests (aws, 9 failures in 7 days):
This pull request and its description were written by Isaac.