Repository navigation
sync: skip files and directories removed during listing - #7605
Merged
Merged
Conversation
fs.WalkDir lists a directory and then reads each child. If a child directory is removed in between (e.g. while sync --watch runs), the walk returned ErrNotExist and the whole sync command failed. Skip such entries; the next listing picks up the removal. A missing top-level path is still an error. Co-authored-by: Isaac <no-reply@databricks.com>
Collaborator
✅ Integration test reportCommit: 32e093a
Top 4 slowest tests (at least 2 minutes):
|
denik
marked this pull request as ready for review
November 13, 2026 18:16
denik
requested review from
andrewnester and
rugpanov
and removed request for
rugpanov
November 13, 2026 18:16
andrewnester
approved these changes
Nov 15, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
sync --watchexits with an error when a directory is removed while it lists files. The integration testTestSyncIncrementalFileOverwritesFolderhits this on Windows (azure and gcp, on main and on several PRs):The test removes
foo/bar.txtand thenfoo.fs.WalkDirinlibs/filesetlists the parent first and then readsfoo. Iffoois removed in between, the walk returnsErrNotExist, and the error ends the whole sync. Windows deletes are slower to settle, so the race shows up there. Users editing files duringsync --watchorbundle sync --watchcan hit the same thing.This is not a recent regression:
recursiveListFileshas walked this way for a long time, and the test has flaked since at least September.Changes
FileSet.recursiveListFilesnow skips entries under the requested path that returnErrNotExist. The next listing iteration sees the removal and deletes the remote copy as before. A requested top-level path that does not exist is still an error.Tests
TestFileSet_SkipsEntryRemovedDuringListing. It uses avfs.Pathwrapper whoseReadDirof a listed directory returnsErrNotExist. Without the fix it fails withopen dir1: file does not exist; with the fix it passes.TestFileSet_MissingRootPathIsAnError, which keeps the existing behavior for a missing top-level path.go test ./libs/fileset ./libs/syncpass.Dashboard
Error: cannot list files: open foo: The system cannot find the file specified.Test: integration/cmd/sync.TestSyncIncrementalFileOverwritesFolder
This pull request and its description were written by Isaac.