Skip to content

filer: retry throttled Files API calls on UC Volumes - #7559

Merged
denik merged 4 commits into
mainfrom
denik/filer-retry-throttling
Nov 12, 2026
Merged

denik merged 4 commits into
mainfrom
denik/filer-retry-throttling

Conversation

@denik

@denik denik commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Changes

Retry Files API calls made by libs/filer.FilesClient (UC Volumes: fs cp/ls/rm/cat/mkdir, bundle uploads to Volumes) on HTTP 408, 429, 502, 503 and 504 and on transient network errors. A call gets up to 10 retries with 1s..10s jittered exponential backoff, and a Retry-After hint is honored.

  • Every files/v2 call (metadata, download, delete, list, mkdir) gets call.WithRetrier(...).
  • UploadFile streams its request body, and files/v2 rejects a retrier on it. The filer retries it itself with ops.Execute: a seekable reader (local files, strings.Reader, bytes.Reader) is rewound to the offset it had when Write was called before each attempt. Each attempt's body is wrapped, so a late read by net/http from an earlier attempt cannot race with the rewind. A non-seekable reader, including a pipe (an *os.File whose Seek fails), still gets one attempt.
  • The multipart upload engine (behind DATABRICKS_EXPERIMENTAL_MULTIPART_UPLOAD) already retries its control-plane calls and is unchanged.

Why

This is a regression from #6018 (2026-07-29). That PR moved the filer from databricks-sdk-go's client.Do to the sdk-go/files/v2 client. The old client retried 429 and 503 automatically. files/v2 retries nothing unless the caller passes a retrier (options/call: "If no retrier is provided, the call is not retried"), and the filer never passed one.

When cloud storage throttles, the Files API returns 429 RESOURCE_EXHAUSTED with the message "AWS S3 is throttling requests; try again later." The caller is expected to retry, so this is not a backend bug. Without retries, a single throttled request fails the whole command.

Decisions:

  • The retrier is the SDK's own core/apiretry.NewRetrier (the one files/v2 uses for its control-plane calls) with a retry cap, and the status set is the same as that path's. The filer's single-shot calls now behave like the multipart engine's calls.
  • The retry cap of 10 keeps the hand-rolled upload retry bounded: unlike non-streaming calls, the upload has no 5-minute SDK timeout, and a whole-call timeout would cut off long uploads.
  • Non-idempotent calls (delete, upload without overwrite) are retried as well, as they were before Add experimental support for large file upload. #6018. If a 5xx response arrives for a request that was in fact applied, the retry can return 404 or 409. A 429 is a rejection, so it does not have this problem.
  • The retry option is passed at each call site. A client-level default retrier (in the SDK, with UploadFile opting out) would be the more robust long-term fix.

Tests

  • New unit tests in libs/filer/files_client_test.go cover: throttled metadata, upload, download and delete calls are retried; a retried upload resends the body from the reader's starting offset; a non-seekable upload is not retried; retries stop after the cap. Without the fix, the retry tests fail.
  • Ran the affected integration tests on AWS: TestFilerRecursiveDelete, TestFsCpDirToDirFileNotOverwritten, TestFsCpFileToDirWithOverwriteFlag, TestFsCpFileToFileWithOverwriteFlag and TestFsCpSourceIsDirectoryButTargetIsFile, all variants. All pass. Throttling did not reproduce during the run.
  • go test ./acceptance -run 'TestAccept/cmd/fs|TestAccept/bundle/libraries' passes.

Dashboard

error verbatim snippet
error/1136 Error: databricks-api error: AWS S3 is throttling requests; try again later.

Flaky tests (aws, 9 failures in 7 days):

This pull request and its description were written by Isaac.

Since the filer moved to the sdk-go files/v2 client, Files API calls are
no longer retried: files/v2 retries nothing unless the caller passes a
retrier. The backend reports throttled cloud storage ("AWS S3 is
throttling requests; try again later.") as 429 RESOURCE_EXHAUSTED and
expects the caller to retry.

Retry 429 and 503 on every files/v2 call the filer makes. The single-shot
upload cannot take a retrier, so the filer retries it itself, rewinding a
seekable reader to its starting offset before each attempt.

Co-authored-by: Isaac <no-reply@databricks.com>
@denik denik added the ai-auto PRs made by AI loops & factories, label Nov 12, 2026
denik and others added 2 commits November 12, 2026 13:55
Co-authored-by: Isaac <no-reply@databricks.com>
…reuse

Use apiretry.NewRetrier with a retry cap, so Retry-After hints and
transient network errors are honored, and retry the same statuses
files/v2 retries for its own control-plane calls (408, 429, 502, 503,
504). Wrap each upload attempt's body so a late read by net/http cannot
race with rewinding the reader for the next attempt.

Co-authored-by: Isaac <no-reply@databricks.com>
@denik
denik requested a review from andrewnester November 12, 2026 14:01
@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Collaborator

✅ Integration test report

Commit: 59079aa

Run: 43265827222

Env 💚​RECOVERED ✅​pass 🙈​skip Time
💚​ aws linux-2core-8gb 1 270 15 10:47
💚​ aws-windows-latest-4core-16gb 1 272 13 7:27
💚​ azure linux-2core-8gb 1 269 15 9:44
💚​ azure-windows-latest-4core-16gb 1 271 13 8:13
💚​ gcp linux-2core-8gb 1 270 15 10:47
💚​ gcp-windows-latest-4core-16gb 1 272 13 6:42
Test Name aws linux-2core-8gb aws-windows-latest-4core-16gb azure linux-2core-8gb azure-windows-latest-4core-16gb gcp linux-2core-8gb gcp-windows-latest-4core-16gb
💚​ TestAccept 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
Top 10 slowest tests (at least 2 minutes):
duration env testname
4:01 gcp linux-2core-8gb TestAccept
3:51 aws linux-2core-8gb TestAccept
3:50 aws-windows-latest-4core-16gb TestAccept
3:46 azure linux-2core-8gb TestAccept
3:43 azure-windows-latest-4core-16gb TestAccept
3:33 gcp-windows-latest-4core-16gb TestAccept
2:45 azure linux-2core-8gb TestImportDirDoesNotOverwrite
2:38 gcp linux-2core-8gb TestFilerRecursiveDelete/workspace_files_extensions
2:26 aws-windows-latest-4core-16gb TestFilerWorkspaceFilesExtensionsDelete
2:10 gcp-windows-latest-4core-16gb TestFsCpDir/dbfs_to_dbfs

@denik
denik marked this pull request as ready for review November 12, 2026 16:32
@denik
denik requested review from a team as code owners November 12, 2026 16:32
@denik
denik requested a review from rauchy November 12, 2026 16:32

@andrewnester andrewnester left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • .nextchanges/cli/7559.md: the entry says "HTTP 429 or 503", but the set is 408/429/502/503/504 plus transient network errors.

Co-authored-by: Isaac <no-reply@databricks.com>
@denik

denik commented Nov 12, 2026

Copy link
Copy Markdown
Contributor Author

Thanks @andrewnester, agreed. .nextchanges/cli/7559.md now lists 408/429/502/503/504 and transient network errors (59079aa). Re-requesting your review.

AI agent (harness_auto, opus5.5, isaac_claude)

@denik
denik requested a review from andrewnester November 12, 2026 17:05
@denik
denik added this pull request to the merge queue Nov 12, 2026
Merged via the queue into main with commit a29c957 Nov 12, 2026
28 checks passed
@denik
denik deleted the denik/filer-retry-throttling branch November 12, 2026 18:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ai-auto PRs made by AI loops & factories,

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants