[denik]

Changes

FilesClient.Write (UC Volumes) now buffers a non-seekable upload body of up to 8 MiB in memory, so the upload is retried on 408/429/502/503/504 like a seekable one. Bodies over 8 MiB are streamed with one attempt, as before.

Why

#7559 retries Files API uploads by rewinding a seekable reader. fs cp from a Volume or from DBFS to a Volume passes the source's download stream to Write, which is not seekable, so a single 503 on the upload still failed the copy. A local source (*os.File) is seekable and already covered.

The errors below are HTTP 503 from the Files API. All failures in the window predate #7556 and #7559 (merged 2026-11-12 ~18:10 UTC). One later PR failure ran on a stacked branch that does not include them. Those PRs cover reads, stats, deletes, mkdir and seekable writes, including the test-setup writes behind most of these failures. This PR closes the remaining gap: remote-to-Volume copies.

Decisions:

• Buffer in the filer, not in cmd/fs, so every caller (fs cp, bundle uploads of generated content) gets it, and the size does not have to be known up front.
• 8 MiB keeps memory bounded per upload (fs cp runs a bounded number of copies at a time). #7556 chose not to buffer multi-GB streams, and this keeps that.
• A retried upload without overwrite can return "already exists" if the first attempt was applied. filer: retry throttled Files API calls on UC Volumes #7559 accepted that for seekable readers, and the same applies here.

Open points for the reviewer:

• Another option is to retry at the copy layer: cpFileToFile would re-open the source with sourceFiler.Read and retry the Write. That covers files of any size with no buffering, but only for fs cp, and cmd/fs would need its own status-code policy. I went with the smaller, filer-local change. Happy to switch.
• Files of 8 MiB or less are now fully downloaded before their upload starts, instead of streaming download into upload. For small files this costs little.

Tests

• Replaced TestFilesClientDoesNotRetryNonSeekableUpload with three tests. TestFilesClientRetriesSmallNonSeekableUpload covers a body of exactly the buffer size and fails without the fix. TestFilesClientRetriesSmallPipeUpload covers an *os.File whose Seek fails. TestFilesClientDoesNotRetryLargeNonSeekableUpload checks a larger body gets one attempt and is sent in full. The tests lower the limit to a few bytes.
• go test -race ./libs/filer/ ./cmd/fs/ and local TestAccept/cmd/fs pass.
• On AWS, integration/cmd/fs TestFsCpDir, TestFsCpDirToDirFileNotOverwritten, TestFsCpFileToFileFileNotOverwritten and TestFsRmEmptyDir pass, all variants. A 503 cannot be forced on cloud.

Dashboard

error	verbatim
error/1140	databricks-api error: Azure is temporarily unavailable; try again later. (17 failures, 13 tests, Azure)
error/1142	databricks-api error: GCP is temporarily unavailable; try again later. (6 failures, GCP)
error/3319	databricks-api error: The service at /api/2.0/fs/files/[PATH] is temporarily unavailable. Please try again later.

Remote-to-Volume tests this PR covers:

• TestFsCpDirToDirFileNotOverwritten/uc-volumes_to_uc-volumes
• TestFsCpDir/uc-volumes_to_uc-volumes
• TestFsCpDirToDirFileNotOverwritten/dbfs_to_uc-volumes
• TestFsCpDirToDirWithOverwriteFlag/dbfs_to_uc-volumes

AI agent (harness_auto).

This pull request and its description were written by Isaac.

[eng-dev-ecosystem-bot]

✅ Integration test report

Commit: 4a696ee

Run: 43461932977

	Env	✅​pass	🙈​skip	Time
✅​	aws linux-2core-8gb	271	15	6:50
✅​	aws-windows-latest-4core-16gb	273	13	5:08
✅​	azure linux-2core-8gb	270	15	6:14
✅​	azure-windows-latest-4core-16gb	272	13	4:51
✅​	gcp linux-2core-8gb	271	15	6:19
✅​	gcp-windows-latest-4core-16gb	273	13	5:27

Top 3 slowest tests (at least 2 minutes):

duration	env	testname
3:26	azure-windows-latest-4core-16gb	TestAccept
3:10	aws-windows-latest-4core-16gb	TestAccept
3:09	gcp-windows-latest-4core-16gb	TestAccept