Repository navigation
acc: resolve cloud workspace auth once per process - #7507
Merged
Merged
Conversation
Every cloud acceptance test created its own SDK workspace client, repeating host metadata, OIDC discovery and the token exchange. When the workspace is briefly slow, those requests time out and the default auth chain reports "cannot configure default credentials", failing all tests that start in that window. Share one client across tests; cache only successful results. Co-authored-by: Isaac <no-reply@databricks.com>
Collaborator
✅ Integration test reportCommit: 6b2e4ae
7 interesting tests: 7 flaky
Top 15 slowest tests (at least 2 minutes):
|
denik
marked this pull request as ready for review
November 11, 2026 21:38
denik
enabled auto-merge
November 11, 2026 21:38
andrewnester
self-requested a review
November 12, 2026 08:58
andrewnester
approved these changes
Nov 12, 2026
This was referenced Nov 12, 2026
tedjt
pushed a commit
to tedjt/cli
that referenced
this pull request
Nov 12, 2026
## Changes Comment-only follow-up to [databricks#7507](databricks#7507), addressing [this review comment](databricks#7507 (comment)). The comment on `cloudWorkspaceState` said auth resolution was not retried. That was inaccurate: the SDK wraps the M2M token exchange in `NewRetryingTokenSource`. Only the OIDC endpoint discovery GET (`getOidcEndpoints` via `refreshClient`, which retries only throttling errors) is left unretried on timeout. The comment now names the discovery request. ## Tests No behavior change; `go build ./acceptance/internal/` passes. _This PR was created by an AI agent (harness_auto, opus5.5, isaac_claude)._ This pull request and its description were written by Isaac. Co-authored-by: Isaac <no-reply@databricks.com>
tedjt
pushed a commit
to tedjt/cli
that referenced
this pull request
Nov 12, 2026
## Changes Go integration tests now share one SDK workspace client per test process. `acc.WorkspaceTest` used to build a new one with `databricks.NewWorkspaceClient()` for every test. ## Why This is the same flaky failure that [databricks#7507](databricks#7507) fixed for acceptance tests, here on the Go integration test path. Each new client repeats OIDC endpoint discovery before its first API call, and the SDK does not retry that request when it times out. If the workspace is slow for a moment, every test starting then fails at its first API call with `default auth: cannot configure default credentials`. The failures since databricks#7507 merged all come from `integration/internal/acc/fixtures.go`. A shared client authenticates once and refreshes its token in the background. The SDK does not cache a failed authentication (`credentialsProvider` is set only on success), so after a transient failure the next call on the shared client tries again. No integration test changes the client's config or the `DATABRICKS_*` auth env. ## Dashboard [error/1224](http://go/cli/dashboard/error/1224?main=1&pr=1&end=2026-11-12T16:00:00Z&last=1d) ``` fixtures.go:111: Error: Received unexpected error: default auth: cannot configure default credentials, please check https://docs.databricks.com/en/dev-tools/auth.html#databricks-client-unified-authentication to configure credentials for your preferred authentication method. Config: host=[DATABRICKS_URL], actions_id_token_request_url=[ACTIONS_URL], actions_id_token_request_token=[TOKEN], client_id=[ID], client_secret=***. Env: DATABRICKS_HOST, ACTIONS_ID_TOKEN_REQUEST_URL, ACTIONS_ID_TOKEN_REQUEST_TOKEN, DATABRICKS_CLIENT_ID, DATABRICKS_CLIENT_SECRET ``` ## Tests Ran locally against AWS: `integration/cmd/fs` (25 tests: the 4 that failed plus others), `integration/libs/filer` (44) and `integration/cmd/workspace` (14). All pass. _This PR was created by an AI agent (harness_auto, opus5.5, isaac_claude)._ This pull request and its description were written by Isaac. Co-authored-by: Isaac <no-reply@databricks.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
Cloud acceptance tests now share one SDK workspace client per test process, instead of each test building its own in
PrepareServerAndClient. Only a successful result (client, current user, workspace id) is cached. If resolving fails, that one test fails and the next test tries again.Why
Every cloud test created a new
databricks.NewWorkspaceClient(), so each of the ~1000 tests in a run repeated host-metadata discovery, OIDC endpoint discovery and the M2M token exchange before its first API call. When the workspace is slow for a few minutes (the failing logs showFailed to resolve host metadata ... request timed outjust before the error), the OIDC or token request times out. The SDK default auth chain throws away each strategy's error and reports only the genericcannot configure default credentials. This error comes from the request visitor, so the API client does not retry it the way it retries a timed-out API call. Every test that starts during that window fails atprepare_server.go"Failed to get current user". Failures come in bursts of 3-8 tests per job across AWS, Azure and GCP, which fits this explanation.With a shared client, auth is resolved once. The token is cached and refreshed in the background, and later API calls go through the SDK's normal retry on timeouts. The CLI subprocess env (
auth.ProcessEnv(cfg)) is the same as before, since every test resolved the same config from the same environment.cfgis only read after this point.Decision: I chose to share the client instead of adding a retry around
GetCurrentUser. It removes thousands of redundant auth round trips per run, and it does not hide real auth misconfiguration: the first test still fails with the real error.Dashboard
error/1224
Tests
Ran locally against AWS:
bundle/config-remote-sync/job_*(8 tests) and the fullbundle/config-remote-syncsuite (54 tests, in parallel, one shared client): all pass. PR CI selects no acceptance tests for this change, so these local runs are the cloud coverage for the shared-client path.This PR was created by an AI agent (harness_auto, opus5.5, isaac_claude).
This pull request and its description were written by Isaac.