Skip to content

Wait out 429 retries in sync integration tests - #7558

Open
denik wants to merge 1 commit into
mainfrom
denik/sync-test-429-timeout
Open

denik wants to merge 1 commit into
mainfrom
denik/sync-test-429-timeout

Conversation

@denik

@denik denik commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Why

The integration/cmd/sync tests fail on all three clouds with:

sync_test.go:114: timed out waiting for sync to complete
runner.go:129: Error running command: Too many requests. Please wait a moment and try again. If the issue persists, consider adjusting your request frequency or reaching out to support for assistance.

The shared test workspace rate-limits POST /api/2.0/workspace-files/import-file when several CI runs upload at once. Debug logs from other tests in the same job show it:

Debug: POST /api/2.0/workspace-files/import-file/[USERNAME]/.../file-a?overwrite=false
< HTTP/2.0 429 Too Many Requests
<   "error_code": "RESOURCE_EXHAUSTED",

The SDK retries 429 with backoff (up to 10s between attempts) for up to 5 minutes. waitForCompletionMarker gave up after 60s and cancelled sync while it was still retrying, so the error printed is the last retried 429. Other tests that upload through import-file see the same 429s and get through on retry. In the dashboard window, only cmd/sync tests hit this error.

The CLI already retries, so nothing changes in production code.

Changes

  • waitForCompletionMarker waits 6 minutes, past the SDK's 5-minute retry window. When nothing is throttled, sync still finishes in a second or two, so passing tests don't get slower.
  • If sync exits before printing the completion event, the test now fails with "sync exited before completing" instead of a JSON decode error on the closed channel.

The timeout was raised before, from 30s to 60s in #3202, for the same load-related flakiness.

Tests

go test ./integration/cmd/sync -run TestSync passes against an AWS test workspace.

This PR's integration run still saw 11 to 32 import-file 429s per job, and every sync test passed on the first attempt. Two of them would have hit the old 60s limit: TestSyncIncrementalFileSync took 265s (azure linux) and one sync test took 156s (gcp windows).

Dashboard

link snippet
error/238 Error: Too many requests. Please wait a moment and try again.
integration/cmd/sync.TestSyncIncrementalFileSync timed out waiting for sync to complete
integration/cmd/sync.TestSyncNestedFolderSync same
integration/cmd/sync.TestSyncIncrementalFileOverwritesFolder same
integration/cmd/sync.TestSyncFullFileSync same

The same error also hits TestSyncIncrementalSyncFileToPythonNotebook, TestSyncIncrementalSyncPythonNotebookToFile, TestSyncIncrementalSyncPythonNotebookDelete, TestSyncNestedFolderDoesntFailOnNonEmptyDirectory and TestSyncNestedSpacePlusAndHashAreEscapedSync.

Written by an AI agent (harness_auto).

This pull request and its description were written by Isaac.

The shared test workspace returns 429 from workspace-files/import-file
when several CI runs upload at once. The SDK retries it for up to 5
minutes, but waitForCompletionMarker gave up after 60s and cancelled
sync mid-retry.

Co-authored-by: Isaac <no-reply@databricks.com>
@denik denik added the ai-auto PRs made by AI loops & factories, label Nov 12, 2026
@github-actions github-actions Bot added the test-only-fix PRs that update failing of flaky tests without changing CLI code. label Nov 12, 2026
@eng-dev-ecosystem-bot

Copy link
Copy Markdown
Collaborator

✅ Integration test report

Commit: 4ab80f2

Run: 43231656405

Env 💚​RECOVERED ✅​pass 🙈​skip Time
💚​ aws linux-2core-8gb 1 270 15 15:37
💚​ aws-windows-latest-4core-16gb 1 272 13 9:00
💚​ azure linux-2core-8gb 1 269 15 17:07
💚​ azure-windows-latest-4core-16gb 1 271 13 10:53
💚​ gcp linux-2core-8gb 1 270 15 14:52
💚​ gcp-windows-latest-4core-16gb 1 272 13 9:30
Test Name aws linux-2core-8gb aws-windows-latest-4core-16gb azure linux-2core-8gb azure-windows-latest-4core-16gb gcp linux-2core-8gb gcp-windows-latest-4core-16gb
💚​ TestAccept 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
Top 30 slowest tests (at least 2 minutes):
duration env testname
5:07 azure-windows-latest-4core-16gb TestAccept
4:32 azure-windows-latest-4core-16gb TestFilerRecursiveDelete/workspace_files_extensions
4:24 azure linux-2core-8gb TestSyncIncrementalFileSync
4:24 gcp linux-2core-8gb TestFilerWorkspaceFilesExtensionsReadDir
3:52 azure linux-2core-8gb TestAccept
3:49 aws linux-2core-8gb TestAccept
3:42 gcp linux-2core-8gb TestAccept
3:37 gcp-windows-latest-4core-16gb TestAccept
3:33 aws linux-2core-8gb TestFilerWorkspaceFilesExtensionsReadDir
3:19 azure-windows-latest-4core-16gb TestFilerWorkspaceFilesExtensionsReadDir
3:15 aws linux-2core-8gb TestFilerWorkspaceFilesExtensionsRead
3:09 aws-windows-latest-4core-16gb TestFilerWorkspaceFilesExtensionsDelete
3:09 aws-windows-latest-4core-16gb TestAccept
2:56 gcp-windows-latest-4core-16gb TestFilerWorkspaceFilesExtensionsReadDir
2:55 azure-windows-latest-4core-16gb TestFilerRecursiveDelete/workspace_files
2:48 gcp linux-2core-8gb TestFilerWorkspaceFilesExtensionsRead
2:44 gcp-windows-latest-4core-16gb TestFilerWorkspaceFilesExtensionsStat
2:43 azure linux-2core-8gb TestFilerWorkspaceFilesExtensionsReadDir
2:35 gcp-windows-latest-4core-16gb TestSyncIncrementalFileSync
2:22 azure linux-2core-8gb TestFilerWorkspaceFilesExtensionsStat
2:21 azure linux-2core-8gb TestFilerReadWrite/workspace_files
2:19 gcp linux-2core-8gb TestFilerWorkspaceFilesExtensionsStat
2:19 aws-windows-latest-4core-16gb TestFilerReadWrite/workspace_files
2:18 gcp linux-2core-8gb TestFilerWorkspaceNotebook/sqlJupyterNotebook.ipynb
2:13 aws-windows-latest-4core-16gb TestImportDirDoesNotOverwrite
2:12 aws linux-2core-8gb TestFilerReadWrite/workspace_files_extensions
2:09 azure-windows-latest-4core-16gb TestExportDir
2:03 azure linux-2core-8gb TestFilerWorkspaceFilesExtensionsDelete
2:02 azure linux-2core-8gb TestFilerWorkspaceNotebook/pythonJupyterNb.ipynb
2:02 aws linux-2core-8gb TestFilerRecursiveDelete/workspace_files_extensions

@denik
denik marked this pull request as ready for review November 12, 2026 14:20
@denik
denik requested review from a team as code owners November 12, 2026 14:20
@denik
denik requested review from rauchy and rclarey November 12, 2026 14:20
@denik
denik enabled auto-merge November 12, 2026 14:20
idelafuente-oc pushed a commit to idelafuente-oc/cli that referenced this pull request Nov 14, 2026
…abricks#7592)

## Why

`TestSyncIncrementalFileSync` and `TestSyncFullFileSync` flake on all
three clouds. Sync uploads the file and prints its completion event, and
then the check that reads it back times out:

```
sync_test.go:161:
    Error:      	Condition never satisfied
runner.go:129: Error running command: context canceled
```

`context canceled` is only the test cleanup stopping `sync --watch`. The
real failure is the 30s `Eventually` in `remoteFileContent` (and the
same 30s in `remoteDirContent`, `objectType` and `language`).

Those reads go through the SDK, which retries throttled (429) requests
for up to 5 minutes. The shared test workspace throttles these sync
tests heavily; see databricks#7558, which handles the same 429s on the upload
side. The timing fits a read that is still retrying: in the failing runs
the test fails about 30s after sync finished, then the in-flight read
returns and the test exits. Failures include the very first read of a
newly created empty file, where stale content can't explain a mismatch.

## Changes

- The four remote-state checks wait up to 6 minutes, past the SDK's
5-minute retry window. When nothing is throttled the first read matches,
so passing runs don't get slower.

This complements databricks#7558 (the completion-marker timeout), and the two
don't touch the same lines.

## Tests

`go test ./integration/cmd/sync -run
'TestSyncFullFileSync|TestSyncIncrementalFileSync|TestSyncNestedFolderSync'`
passes against an AWS test workspace.

## Dashboard

| link | snippet |
|---|---|
|
[error/705](http://go/cli/dashboard/error/705?main=1&pr=1&end=2026-11-13T09:00:00Z&last=7d)
| `Error: context canceled` after `sync_test.go:161: Condition never
satisfied` |
|
[integration/cmd/sync.TestSyncIncrementalFileSync](http://go/cli/dashboard/test/integration/cmd/sync.TestSyncIncrementalFileSync?main=1&pr=1&end=2026-11-13T09:00:00Z&last=7d)
| same, 9 failures in 7 days |
|
[integration/cmd/sync.TestSyncFullFileSync](http://go/cli/dashboard/test/integration/cmd/sync.TestSyncFullFileSync?main=1&pr=1&end=2026-11-13T09:00:00Z&last=7d)
| same, 5 failures in 7 days |

_Written by an AI agent (harness_auto)._

This pull request and its description were written by Isaac.

Co-authored-by: Isaac <no-reply@databricks.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ai-auto PRs made by AI loops & factories, test-only-fix PRs that update failing of flaky tests without changing CLI code.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants