Repository navigation
Conversation
The shared test workspace returns 429 from workspace-files/import-file when several CI runs upload at once. The SDK retries it for up to 5 minutes, but waitForCompletionMarker gave up after 60s and cancelled sync mid-retry. Co-authored-by: Isaac <no-reply@databricks.com>
Collaborator
✅ Integration test reportCommit: 4ab80f2
Top 30 slowest tests (at least 2 minutes):
|
denik
marked this pull request as ready for review
November 12, 2026 14:20
denik
enabled auto-merge
November 12, 2026 14:20
idelafuente-oc
pushed a commit
to idelafuente-oc/cli
that referenced
this pull request
Nov 14, 2026
…abricks#7592) ## Why `TestSyncIncrementalFileSync` and `TestSyncFullFileSync` flake on all three clouds. Sync uploads the file and prints its completion event, and then the check that reads it back times out: ``` sync_test.go:161: Error: Condition never satisfied runner.go:129: Error running command: context canceled ``` `context canceled` is only the test cleanup stopping `sync --watch`. The real failure is the 30s `Eventually` in `remoteFileContent` (and the same 30s in `remoteDirContent`, `objectType` and `language`). Those reads go through the SDK, which retries throttled (429) requests for up to 5 minutes. The shared test workspace throttles these sync tests heavily; see databricks#7558, which handles the same 429s on the upload side. The timing fits a read that is still retrying: in the failing runs the test fails about 30s after sync finished, then the in-flight read returns and the test exits. Failures include the very first read of a newly created empty file, where stale content can't explain a mismatch. ## Changes - The four remote-state checks wait up to 6 minutes, past the SDK's 5-minute retry window. When nothing is throttled the first read matches, so passing runs don't get slower. This complements databricks#7558 (the completion-marker timeout), and the two don't touch the same lines. ## Tests `go test ./integration/cmd/sync -run 'TestSyncFullFileSync|TestSyncIncrementalFileSync|TestSyncNestedFolderSync'` passes against an AWS test workspace. ## Dashboard | link | snippet | |---|---| | [error/705](http://go/cli/dashboard/error/705?main=1&pr=1&end=2026-11-13T09:00:00Z&last=7d) | `Error: context canceled` after `sync_test.go:161: Condition never satisfied` | | [integration/cmd/sync.TestSyncIncrementalFileSync](http://go/cli/dashboard/test/integration/cmd/sync.TestSyncIncrementalFileSync?main=1&pr=1&end=2026-11-13T09:00:00Z&last=7d) | same, 9 failures in 7 days | | [integration/cmd/sync.TestSyncFullFileSync](http://go/cli/dashboard/test/integration/cmd/sync.TestSyncFullFileSync?main=1&pr=1&end=2026-11-13T09:00:00Z&last=7d) | same, 5 failures in 7 days | _Written by an AI agent (harness_auto)._ This pull request and its description were written by Isaac. Co-authored-by: Isaac <no-reply@databricks.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The
integration/cmd/synctests fail on all three clouds with:The shared test workspace rate-limits
POST /api/2.0/workspace-files/import-filewhen several CI runs upload at once. Debug logs from other tests in the same job show it:The SDK retries 429 with backoff (up to 10s between attempts) for up to 5 minutes.
waitForCompletionMarkergave up after 60s and cancelled sync while it was still retrying, so the error printed is the last retried 429. Other tests that upload through import-file see the same 429s and get through on retry. In the dashboard window, onlycmd/synctests hit this error.The CLI already retries, so nothing changes in production code.
Changes
waitForCompletionMarkerwaits 6 minutes, past the SDK's 5-minute retry window. When nothing is throttled, sync still finishes in a second or two, so passing tests don't get slower.The timeout was raised before, from 30s to 60s in #3202, for the same load-related flakiness.
Tests
go test ./integration/cmd/sync -run TestSyncpasses against an AWS test workspace.This PR's integration run still saw 11 to 32 import-file 429s per job, and every sync test passed on the first attempt. Two of them would have hit the old 60s limit: TestSyncIncrementalFileSync took 265s (azure linux) and one sync test took 156s (gcp windows).
Dashboard
Error: Too many requests. Please wait a moment and try again.timed out waiting for sync to completeThe same error also hits TestSyncIncrementalSyncFileToPythonNotebook, TestSyncIncrementalSyncPythonNotebookToFile, TestSyncIncrementalSyncPythonNotebookDelete, TestSyncNestedFolderDoesntFailOnNonEmptyDirectory and TestSyncNestedSpacePlusAndHashAreEscapedSync.
Written by an AI agent (harness_auto).
This pull request and its description were written by Isaac.