Skip to content

acc: resolve cloud workspace auth once per process - #7507

Merged
denik merged 1 commit into
mainfrom
acc-share-cloud-workspace-client
Nov 12, 2026
Merged

denik merged 1 commit into
mainfrom
acc-share-cloud-workspace-client

Conversation

@denik

@denik denik commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Changes

Cloud acceptance tests now share one SDK workspace client per test process, instead of each test building its own in PrepareServerAndClient. Only a successful result (client, current user, workspace id) is cached. If resolving fails, that one test fails and the next test tries again.

Why

Every cloud test created a new databricks.NewWorkspaceClient(), so each of the ~1000 tests in a run repeated host-metadata discovery, OIDC endpoint discovery and the M2M token exchange before its first API call. When the workspace is slow for a few minutes (the failing logs show Failed to resolve host metadata ... request timed out just before the error), the OIDC or token request times out. The SDK default auth chain throws away each strategy's error and reports only the generic cannot configure default credentials. This error comes from the request visitor, so the API client does not retry it the way it retries a timed-out API call. Every test that starts during that window fails at prepare_server.go "Failed to get current user". Failures come in bursts of 3-8 tests per job across AWS, Azure and GCP, which fits this explanation.

With a shared client, auth is resolved once. The token is cached and refreshed in the background, and later API calls go through the SDK's normal retry on timeouts. The CLI subprocess env (auth.ProcessEnv(cfg)) is the same as before, since every test resolved the same config from the same environment. cfg is only read after this point.

Decision: I chose to share the client instead of adding a retry around GetCurrentUser. It removes thousands of redundant auth round trips per run, and it does not hide real auth misconfiguration: the first test still fails with the real error.

Dashboard

error/1224

prepare_server.go:124:
    Error:      Received unexpected error:
                failed during request visitor: default auth: cannot configure default credentials, please check https://docs.databricks.com/en/dev-tools/auth.html#databricks-client-unified-authentication to configure credentials for your preferred authentication method. Config: host=[DATABRICKS_URL], actions_id_token_request_url=[ACTIONS_URL], actions_id_token_request_token=[TOKEN], client_id=[ID], client_secret=***. Env: DATABRICKS_HOST, ACTIONS_ID_TOKEN_REQUEST_URL, ACTIONS_ID_TOKEN_REQUEST_TOKEN, DATABRICKS_CLIENT_ID, DATABRICKS_CLIENT_SECRET
    Messages:   Failed to get current user

Tests

Ran locally against AWS: bundle/config-remote-sync/job_* (8 tests) and the full bundle/config-remote-sync suite (54 tests, in parallel, one shared client): all pass. PR CI selects no acceptance tests for this change, so these local runs are the cloud coverage for the shared-client path.

This PR was created by an AI agent (harness_auto, opus5.5, isaac_claude).

This pull request and its description were written by Isaac.

Every cloud acceptance test created its own SDK workspace client, repeating
host metadata, OIDC discovery and the token exchange. When the workspace is
briefly slow, those requests time out and the default auth chain reports
"cannot configure default credentials", failing all tests that start in that
window. Share one client across tests; cache only successful results.

Co-authored-by: Isaac <no-reply@databricks.com>
@denik denik added the ai-auto PRs made by AI loops & factories, label Nov 11, 2026
@github-actions github-actions Bot added the test-only-fix PRs that update failing of flaky tests without changing CLI code. label Nov 11, 2026
@eng-dev-ecosystem-bot

Copy link
Copy Markdown
Collaborator

✅ Integration test report

Commit: 6b2e4ae

Run: 43091758659

Env 🔄​flaky ✅​pass 🙈​skip Time
✅​ aws linux-2core-8gb 271 15 11:04
✅​ aws-windows-latest-4core-16gb 273 13 6:29
✅​ azure linux-2core-8gb 270 15 10:51
✅​ azure-windows-latest-4core-16gb 272 13 7:29
✅​ gcp linux-2core-8gb 271 15 10:07
🔄​ gcp-windows-latest-4core-16gb 7 266 13 35:13
7 interesting tests: 7 flaky
Test Name gcp-windows-latest-4core-16gb
🔄​ TestFsCpDirToDirFileNotOverwritten/dbfs_to_dbfs 🔄​f
🔄​ TestFsCpDirToDirFileNotOverwritten/dbfs_to_local 🔄​f
🔄​ TestFsCpDirToDirFileNotOverwritten/local_to_dbfs 🔄​f
🔄​ TestFsCpDirToDirFileNotOverwritten/uc-volumes_to_dbfs 🔄​f
🔄​ TestSyncIncrementalFileSync 🔄​f
🔄​ TestFilerReadWrite 🔄​f
🔄​ TestFilerReadWrite/dbfs 🔄​f
Top 15 slowest tests (at least 2 minutes):
duration env testname
3:52 azure linux-2core-8gb TestAccept
3:48 gcp linux-2core-8gb TestAccept
3:47 aws linux-2core-8gb TestAccept
3:40 gcp-windows-latest-4core-16gb TestAccept
3:29 aws-windows-latest-4core-16gb TestAccept
3:28 azure-windows-latest-4core-16gb TestAccept
3:24 azure-windows-latest-4core-16gb TestFilerWorkspaceFilesExtensionsStat
3:22 gcp-windows-latest-4core-16gb TestFsCpDirToDirWithOverwriteFlag/dbfs_to_local
2:57 gcp-windows-latest-4core-16gb TestFsCpDirToDirFileNotOverwritten/local_to_dbfs
2:45 gcp-windows-latest-4core-16gb TestFilerWorkspaceFilesExtensionsReadDir
2:32 gcp-windows-latest-4core-16gb TestFsCpDir/dbfs_to_uc-volumes
2:32 gcp-windows-latest-4core-16gb TestFsCpDir/dbfs_to_dbfs
2:27 gcp-windows-latest-4core-16gb TestImportDir
2:20 aws linux-2core-8gb TestFilerRecursiveDelete/workspace_files
2:19 gcp-windows-latest-4core-16gb TestFilerReadWrite/workspace_files

@denik
denik marked this pull request as ready for review November 11, 2026 21:38
@denik
denik requested a review from a team as a code owner November 11, 2026 21:38
@denik
denik requested a review from anton-107 November 11, 2026 21:38
@denik
denik enabled auto-merge November 11, 2026 21:38
@denik denik changed the title acceptance: resolve cloud workspace auth once per process acc: resolve cloud workspace auth once per process Nov 11, 2026
@andrewnester
andrewnester self-requested a review November 12, 2026 08:58
Comment thread acceptance/internal/prepare_server.go Resolved
@denik
denik added this pull request to the merge queue Nov 12, 2026
Merged via the queue into main with commit 30a5fa3 Nov 12, 2026
28 checks passed
@denik
denik deleted the acc-share-cloud-workspace-client branch November 12, 2026 10:07
tedjt pushed a commit to tedjt/cli that referenced this pull request Nov 12, 2026
## Changes

Comment-only follow-up to
[databricks#7507](databricks#7507), addressing [this
review
comment](databricks#7507 (comment)).

The comment on `cloudWorkspaceState` said auth resolution was not
retried. That was inaccurate: the SDK wraps the M2M token exchange in
`NewRetryingTokenSource`. Only the OIDC endpoint discovery GET
(`getOidcEndpoints` via `refreshClient`, which retries only throttling
errors) is left unretried on timeout. The comment now names the
discovery request.

## Tests

No behavior change; `go build ./acceptance/internal/` passes.

_This PR was created by an AI agent (harness_auto, opus5.5,
isaac_claude)._

This pull request and its description were written by Isaac.

Co-authored-by: Isaac <no-reply@databricks.com>
tedjt pushed a commit to tedjt/cli that referenced this pull request Nov 12, 2026
## Changes

Go integration tests now share one SDK workspace client per test
process. `acc.WorkspaceTest` used to build a new one with
`databricks.NewWorkspaceClient()` for every test.

## Why

This is the same flaky failure that
[databricks#7507](databricks#7507) fixed for
acceptance tests, here on the Go integration test path. Each new client
repeats OIDC endpoint discovery before its first API call, and the SDK
does not retry that request when it times out. If the workspace is slow
for a moment, every test starting then fails at its first API call with
`default auth: cannot configure default credentials`. The failures since
databricks#7507 merged all come from `integration/internal/acc/fixtures.go`.

A shared client authenticates once and refreshes its token in the
background. The SDK does not cache a failed authentication
(`credentialsProvider` is set only on success), so after a transient
failure the next call on the shared client tries again. No integration
test changes the client's config or the `DATABRICKS_*` auth env.

## Dashboard


[error/1224](http://go/cli/dashboard/error/1224?main=1&pr=1&end=2026-11-12T16:00:00Z&last=1d)

```
fixtures.go:111:
    Error:      Received unexpected error:
                default auth: cannot configure default credentials, please check https://docs.databricks.com/en/dev-tools/auth.html#databricks-client-unified-authentication to configure credentials for your preferred authentication method. Config: host=[DATABRICKS_URL], actions_id_token_request_url=[ACTIONS_URL], actions_id_token_request_token=[TOKEN], client_id=[ID], client_secret=***. Env: DATABRICKS_HOST, ACTIONS_ID_TOKEN_REQUEST_URL, ACTIONS_ID_TOKEN_REQUEST_TOKEN, DATABRICKS_CLIENT_ID, DATABRICKS_CLIENT_SECRET
```

## Tests

Ran locally against AWS: `integration/cmd/fs` (25 tests: the 4 that
failed plus others), `integration/libs/filer` (44) and
`integration/cmd/workspace` (14). All pass.

_This PR was created by an AI agent (harness_auto, opus5.5,
isaac_claude)._

This pull request and its description were written by Isaac.

Co-authored-by: Isaac <no-reply@databricks.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ai-auto PRs made by AI loops & factories, test-only-fix PRs that update failing of flaky tests without changing CLI code.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants