Skip to content

Record the profile auth docker configure selects for Docker - #7383

Open
yolocs wants to merge 1 commit into
mainfrom
stack/docker-shared-workspace-profiles
Open

yolocs wants to merge 1 commit into
mainfrom
stack/docker-shared-workspace-profiles

Conversation

@yolocs

@yolocs yolocs commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

Changes

databricks auth docker configure <profile> no longer fails when several profiles match the selected workspace. It records the chosen profile in its .databrickscfg section with docker_credential_helper = true and removes the mark from the workspace's other profiles. Running it with another profile switches the selection:

Warning: Docker credential helper for 1234567890123456.container.us-west-2.cloud.databricks.com switched from profile dev to prod
Configured Docker credential helper for 1234567890123456.container.us-west-2.cloud.databricks.com

databricks auth docker token (the Docker credential helper) picks the profile for a registry like this:

  • A recorded profile wins. If it is no longer a supported profile, the helper fails with that profile's error instead of falling back to another identity.
  • With no recorded profile, a single supported match is used, as before. This covers setups configured with older CLI versions.
  • Otherwise it fails with ... Select the profile for Docker with: databricks auth docker configure <profile-name>.

Saving a profile with a different host or workspace ID, for example auth login --profile prod --host <other-workspace> or databricks configure --host, drops its mark, so the mark cannot follow the profile to another workspace.

databricks auth docker host --profile P reports the helper as configured only if Docker's config.json selects it for the registry, it is on PATH, and it would use P.

Why

Docker gives the credential helper only the registry host, which encodes the workspace ID, so the helper cannot tell which of several matching profiles was meant. auth login now saves workspace_id on every profile, so two logins to one workspace were enough to block auth docker configure. Recording the choice at configure time means docker push uses the profile that was configured, whatever the order in .databrickscfg, and the helper never switches identity on its own.

The mark lives on the profile section, so it follows DATABRICKS_CONFIG_FILE, is removed by auth logout --delete, and survives re-logging in to the same workspace. The Go SDK ignores it because it reads only known attributes. Docker's config.json cannot hold it, because docker login rewrites that file and drops unknown keys.

This replaces the "remove the duplicate profile" hint that #7363 added in v1.25.0.

Follow-up: matching still compares only the workspace ID and loopback vs remote host, so a profile on another Databricks deployment with the same workspace ID also matches.

Tests

  • Unit: ProfileForRegistry (recorded profile wins, unsupported recorded profile fails, several recorded, none recorded), RecordProfile (first record, switch, already recorded, stale marks on unsupported profiles cleared, other workspaces untouched, round trip with ProfileForRegistry), SetDockerCredentialHelperProfile (including dotted section names), SaveToProfile dropping the mark when the host or workspace ID changes, the profile loader, and Inspect (Docker config, PATH and profile selection are all required).
  • Acceptance: cmd/auth/docker/{configure,configure-errors,token,help}. Configure switches to another profile and back, with host reporting which profile Docker uses, and prints the resulting .databrickscfg. Token errors when several profiles match and none is recorded, uses the recorded profile over an earlier logged-in one, and errors when several are recorded.
  • go test ./cmd/... ./libs/dockercredentials/... ./libs/auth/... ./libs/databrickscfg/... and go test ./acceptance -run 'TestAccept/(cmd|air|auth)/' pass, including air/image-push, which parses the auth docker configure output. Lint, deadcode, whitespace and changelog checks pass.

This pull request and its description were written by Isaac.

@yolocs
yolocs force-pushed the stack/docker-shared-workspace-profiles branch from 849da49 to 71222b5 Compare November 5, 2026 21:25
@github-actions

github-actions Bot commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

Commit: 68233ed45 | Workflow run

Regression test report

Selected 4 changed tests (limit=20, 0 not selected)

4 tests: regression=3 unreleased=0 coverage=1 failing=0
test branch main (db52460) latest (v1.26.0)
TestAccept/cmd/auth/docker/configure ✅ ❌ ❌
TestAccept/cmd/auth/docker/configure-errors ✅ ✅ ✅
TestAccept/cmd/auth/docker/help ✅ ❌ ❌
TestAccept/cmd/auth/docker/token ✅ ❌ ❌
TestAccept/cmd/auth/docker/configure: branch ✅ | main (db52460) ❌ | latest (v1.26.0) ❌

main (db52460):

acceptance_test.go:1374: Diff:
        --- cmd/auth/docker/configure/output.txt
        +++ /tmp/TestAcceptcmdauthdockerconfigure3114405928/002/output.txt
        @@ -61,10 +61,9 @@
         [DEFAULT]
         
         [docker-test]
        -host                     = [DATABRICKS_URL]
        -auth_type                = databricks-cli
        -workspace_id             = [WORKSPACE_ID]
        -docker_credential_helper = true
        +host         = [DATABRICKS_URL]
        +auth_type    = databricks-cli
        +workspace_id = [WORKSPACE_ID]
         
         >>> databricks auth docker host --profile docker-test --output json
         {
        @@ -159,48 +158,6 @@
         === Profiles sharing the workspace
         
         >>> databricks auth docker configure shared
        -Warning: Docker credential helper for [WORKSPACE_ID].container.us-west-2.localhost switched from profile docker-test to shared
        -Configured Docker credential helper for [WORKSPACE_ID].container.us-west-2.localhost
        -Updated Docker config: [TEST_TMP_DIR]/home/docker/config.json
        -Installed Docker credential helper: [TEST_TMP_DIR]/home/bin/docker-credential-databricks
        +Error: multiple Databricks profiles match workspace ID [WORKSPACE_ID]: docker-test and shared. Docker credential helper requires exactly one profile per workspace. Remove the duplicate profile with: databricks auth logout --profile <profile-name> --delete
         
        ->>> databricks auth docker host --profile docker-test
        -Registry host: [WORKSPACE_ID].container.us-west-2.localhost
        -Credential helper configured: NO
        -
        ->>> databricks auth docker host --profile shared
        -Registry host: [WORKSPACE_ID].container.us-west-2.localhost
        -Credential helper configured: YES
        -
        ->>> databricks auth docker configure docker-test
        -Warning: Docker credential helper for [WORKSPACE_ID].container.us-west-2.localhost switched from profile shared to docker-test
        -Configured Docker credential helper for [WORKSPACE_ID].container.us-west-2.localhost
        -Updated Docker config: [TEST_TMP_DIR]/home/docker/config.json
        -Installed Docker credential helper: [TEST_TMP_DIR]/home/bin/docker-credential-databricks
        -
        ->>> print_requests.py --get --oneline //api
        -{"headers": {"X-Databricks-Workspace-Id": ["[WORKSPACE_ID]"]}, "method": "GET", "path": "/api/2.1/unity-catalog/metastore_summary"}
        -{"headers": {"X-Databricks-Workspace-Id": ["[WORKSPACE_ID]"]}, "method": "GET", "path": "/api/2.1/unity-catalog/metastore_summary"}
        -{"headers": {"X-Databricks-Workspace-Id": ["[WORKSPACE_ID]"]}, "method": "GET", "path": "/api/2.1/unity-catalog/metastore_summary"}
        -{"headers": {"X-Databricks-Workspace-Id": ["[WORKSPACE_ID]"]}, "method": "GET", "path": "/api/2.1/unity-catalog/metastore_summary"}
        -
        ->>> cat [TEST_TMP_DIR]/home/.databrickscfg
        -; The profile defined in the DEFAULT section is to be used as a fallback when no profile is explicitly specified.
        -[DEFAULT]
        -
        -[docker-test]
        -host                     = [DATABRICKS_URL]
        -auth_type                = databricks-cli
        -workspace_id             = [WORKSPACE_ID]
        -docker_credential_helper = true
        -
        -[shared]
        -host         = [DATABRICKS_URL]
        -auth_type    = databricks-cli
        -workspace_id = [WORKSPACE_ID]
        -
        ->>> docker-credential-databricks get
        -{
        -  "Secret": "oauth-token",
        -  "Username": "oauthtoken"
        -}
        +Exit code: 1
        
    acceptance_test.go:1152: 
        LOG.configure-again: Configured Docker credential helper for 900800700600.container.us-west-2.localhost
        LOG.configure-again: Updated Docker config: /tmp/TestAcceptcmdauthdockerconfigure3114405928/001/home/docker/config.json
        LOG.configure-again: Installed Docker credential helper: /tmp/TestAcceptcmdauthdockerconfigure3114405928/001/home/bin/docker-credential-databricks

latest (v1.26.0):

acceptance_test.go:1374: Diff:
        --- cmd/auth/docker/configure/output.txt
        +++ /tmp/TestAcceptcmdauthdockerconfigure2483455077/002/output.txt
        @@ -61,10 +61,9 @@
         [DEFAULT]
         
         [docker-test]
        -host                     = [DATABRICKS_URL]
        -auth_type                = databricks-cli
        -workspace_id             = [WORKSPACE_ID]
        -docker_credential_helper = true
        +host         = [DATABRICKS_URL]
        +auth_type    = databricks-cli
        +workspace_id = [WORKSPACE_ID]
         
         >>> databricks auth docker host --profile docker-test --output json
         {
        @@ -159,48 +158,6 @@
         === Profiles sharing the workspace
         
         >>> databricks auth docker configure shared
        -Warning: Docker credential helper for [WORKSPACE_ID].container.us-west-2.localhost switched from profile docker-test to shared
        -Configured Docker credential helper for [WORKSPACE_ID].container.us-west-2.localhost
        -Updated Docker config: [TEST_TMP_DIR]/home/docker/config.json
        -Installed Docker credential helper: [TEST_TMP_DIR]/home/bin/docker-credential-databricks
        +Error: multiple Databricks profiles match workspace ID [WORKSPACE_ID]: docker-test and shared. Docker credential helper requires exactly one profile per workspace. Remove the duplicate profile with: databricks auth logout --profile <profile-name> --delete
         
        ->>> databricks auth docker host --profile docker-test
        -Registry host: [WORKSPACE_ID].container.us-west-2.localhost
        -Credential helper configured: NO
        -
        ->>> databricks auth docker host --profile shared
        -Registry host: [WORKSPACE_ID].container.us-west-2.localhost
        -Credential helper configured: YES
        -
        ->>> databricks auth docker configure docker-test
        -Warning: Docker credential helper for [WORKSPACE_ID].container.us-west-2.localhost switched from profile shared to docker-test
        -Configured Docker credential helper for [WORKSPACE_ID].container.us-west-2.localhost
        -Updated Docker config: [TEST_TMP_DIR]/home/docker/config.json
        -Installed Docker credential helper: [TEST_TMP_DIR]/home/bin/docker-credential-databricks
        -
        ->>> print_requests.py --get --oneline //api
        -{"headers": {"X-Databricks-Workspace-Id": ["[WORKSPACE_ID]"]}, "method": "GET", "path": "/api/2.1/unity-catalog/metastore_summary"}
        -{"headers": {"X-Databricks-Workspace-Id": ["[WORKSPACE_ID]"]}, "method": "GET", "path": "/api/2.1/unity-catalog/metastore_summary"}
        -{"headers": {"X-Databricks-Workspace-Id": ["[WORKSPACE_ID]"]}, "method": "GET", "path": "/api/2.1/unity-catalog/metastore_summary"}
        -{"headers": {"X-Databricks-Workspace-Id": ["[WORKSPACE_ID]"]}, "method": "GET", "path": "/api/2.1/unity-catalog/metastore_summary"}
        -
        ->>> cat [TEST_TMP_DIR]/home/.databrickscfg
        -; The profile defined in the DEFAULT section is to be used as a fallback when no profile is explicitly specified.
        -[DEFAULT]
        -
        -[docker-test]
        -host                     = [DATABRICKS_URL]
        -auth_type                = databricks-cli
        -workspace_id             = [WORKSPACE_ID]
        -docker_credential_helper = true
        -
        -[shared]
        -host         = [DATABRICKS_URL]
        -auth_type    = databricks-cli
        -workspace_id = [WORKSPACE_ID]
        -
        ->>> docker-credential-databricks get
        -{
        -  "Secret": "oauth-token",
        -  "Username": "oauthtoken"
        -}
        +Exit code: 1
        
    acceptance_test.go:1152: 
        LOG.configure-again: Configured Docker credential helper for 900800700600.container.us-west-2.localhost
        LOG.configure-again: Updated Docker config: /tmp/TestAcceptcmdauthdockerconfigure2483455077/001/home/docker/config.json
        LOG.configure-again: Installed Docker credential helper: /tmp/TestAcceptcmdauthdockerconfigure2483455077/001/home/bin/docker-credential-databricks
FAIL
TestAccept/cmd/auth/docker/help: branch ✅ | main (db52460) ❌ | latest (v1.26.0) ❌

main (db52460):

acceptance_test.go:1374: Diff:
        --- cmd/auth/docker/help/output.txt
        +++ /tmp/TestAcceptcmdauthdockerhelp2931747289/002/output.txt
        @@ -65,14 +65,11 @@
         This command installs docker-credential-databricks and configures Docker to use
         it for the selected workspace's Artifact Registry host. If the selected profile
         does not already include a workspace_id, the command resolves and saves it so
        -the Docker helper can map the registry host back to the profile. The command
        -also marks the profile with docker_credential_helper = true, so the helper uses
        -it when other profiles match the same workspace; run the command with another
        -profile to switch. The registry region is inferred from the workspace's
        -metastore. Select the workspace with [PROFILE] or --profile; --host,
        ---account-id, and --workspace-id are not supported. The deprecated --region
        -flag is retained for compatibility; omit it because it will be fully removed in
        -the next release.
        +the Docker helper can map the registry host back to the profile. The registry
        +region is inferred from the workspace's metastore. Select the workspace with
        +[PROFILE] or --profile; --host, --account-id, and --workspace-id are not
        +supported. The deprecated --region flag is retained for compatibility; omit it
        +because it will be fully removed in the next release.
         
         Usage:
           databricks auth docker configure [PROFILE] [flags]
        @@ -93,9 +90,7 @@
         >>> [CLI] auth docker host --help
         (Experimental) Show the Databricks Artifact Registry host and Docker credential helper status for a profile.
         
        -The credential helper counts as configured only if Docker's config.json selects
        -it for the registry and the helper uses this profile. The --profile flag is
        -required.
        +The --profile flag is required.
         
         Usage:
           databricks auth docker host [flags]

latest (v1.26.0): same output as main (db52460)

TestAccept/cmd/auth/docker/token: branch ✅ | main (db52460) ❌ | latest (v1.26.0) ❌

main (db52460):

acceptance_test.go:1374: Diff:
        --- cmd/auth/docker/token/output.txt
        +++ /tmp/TestAcceptcmdauthdockertoken1289080131/002/output.txt
        @@ -54,17 +54,11 @@
         === Multiple profiles for the workspace
         
         >>> musterr [CLI] auth docker token
        -Error: multiple Databricks profiles match workspace ID [WORKSPACE_ID]: docker-test, duplicate. Select the profile for Docker with: databricks auth docker configure <profile-name>
        +Error: multiple Databricks profiles match workspace ID [WORKSPACE_ID]: docker-test and duplicate. Docker credential helper requires exactly one profile per workspace. Remove the duplicate profile with: databricks auth logout --profile <profile-name> --delete
         
         === Recorded profile for the workspace
         
         >>> [CLI] auth docker token --no-force-refresh
        -{
        -  "Secret": "duplicate-access-token",
        -  "Username": "oauthtoken"
        -}
        +Error: multiple Databricks profiles match workspace ID [WORKSPACE_ID]: docker-test and duplicate. Docker credential helper requires exactly one profile per workspace. Remove the duplicate profile with: databricks auth logout --profile <profile-name> --delete
         
        -=== Several recorded profiles for the workspace
        -
        ->>> musterr [CLI] auth docker token
        -Error: multiple Databricks profiles match workspace ID [WORKSPACE_ID]: duplicate, also-recorded. Select the profile for Docker with: databricks auth docker configure <profile-name>
        +Exit code: 1
        
FAIL

latest (v1.26.0):

acceptance_test.go:1374: Diff:
        --- cmd/auth/docker/token/output.txt
        +++ /tmp/TestAcceptcmdauthdockertoken2816802535/002/output.txt
        @@ -54,17 +54,11 @@
         === Multiple profiles for the workspace
         
         >>> musterr [CLI] auth docker token
        -Error: multiple Databricks profiles match workspace ID [WORKSPACE_ID]: docker-test, duplicate. Select the profile for Docker with: databricks auth docker configure <profile-name>
        +Error: multiple Databricks profiles match workspace ID [WORKSPACE_ID]: docker-test and duplicate. Docker credential helper requires exactly one profile per workspace. Remove the duplicate profile with: databricks auth logout --profile <profile-name> --delete
         
         === Recorded profile for the workspace
         
         >>> [CLI] auth docker token --no-force-refresh
        -{
        -  "Secret": "duplicate-access-token",
        -  "Username": "oauthtoken"
        -}
        +Error: multiple Databricks profiles match workspace ID [WORKSPACE_ID]: docker-test and duplicate. Docker credential helper requires exactly one profile per workspace. Remove the duplicate profile with: databricks auth logout --profile <profile-name> --delete
         
        -=== Several recorded profiles for the workspace
        -
        ->>> musterr [CLI] auth docker token
        -Error: multiple Databricks profiles match workspace ID [WORKSPACE_ID]: duplicate, also-recorded. Select the profile for Docker with: databricks auth docker configure <profile-name>
        +Exit code: 1

@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Nov 5, 2026 •

Copy link
Copy Markdown
Collaborator

✅ Integration test report

Commit: 68233ed

Run: 43323091086

Env 🔄​flaky 💚​RECOVERED ✅​pass 🙈​skip Time
💚​ aws linux-2core-8gb 1 270 19 13:05
🔄​ aws-windows-latest-4core-16gb 1 1 271 17 14:00
🔄​ azure linux-2core-8gb 1 1 268 19 19:45
💚​ azure-windows-latest-4core-16gb 1 271 17 13:00
🔄​ gcp linux-2core-8gb 1 1 269 19 14:09
💚​ gcp-windows-latest-4core-16gb 1 272 17 10:25
Test Name aws linux-2core-8gb aws-windows-latest-4core-16gb azure linux-2core-8gb azure-windows-latest-4core-16gb gcp linux-2core-8gb gcp-windows-latest-4core-16gb
💚​ TestAccept 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
🔄​ TestDashboardAssumptions_WorkspaceImport ✅​p ✅​p 🔄​f ✅​p ✅​p ✅​p
🔄​ TestSyncIncrementalSyncPythonNotebookToFile ✅​p 🔄​f ✅​p ✅​p 🔄​f ✅​p
Top 34 slowest tests (at least 2 minutes):
duration env testname
6:23 azure linux-2core-8gb TestFilerWorkspaceFilesExtensionsReadDir
5:18 aws linux-2core-8gb TestFilerWorkspaceFilesExtensionsReadDir
4:38 azure-windows-latest-4core-16gb TestExportDir
4:17 aws-windows-latest-4core-16gb TestImportDirWithOverwriteFlag
4:15 gcp linux-2core-8gb TestFilerWorkspaceFilesExtensionsDelete
4:00 aws-windows-latest-4core-16gb TestAccept
3:58 gcp-windows-latest-4core-16gb TestImportDirWithOverwriteFlag
3:55 aws linux-2core-8gb TestAccept
3:52 gcp linux-2core-8gb TestAccept
3:50 azure linux-2core-8gb TestAccept
3:43 azure-windows-latest-4core-16gb TestAccept
3:39 azure linux-2core-8gb TestFilerWorkspaceFilesExtensionsDelete
3:38 azure-windows-latest-4core-16gb TestFilerWorkspaceFilesExtensionsRead
3:27 azure-windows-latest-4core-16gb TestImportDir
3:16 gcp-windows-latest-4core-16gb TestFilerWorkspaceFilesExtensionsReadDir
3:15 gcp-windows-latest-4core-16gb TestAccept
3:13 aws-windows-latest-4core-16gb TestFilerWorkspaceFilesExtensionsStat
2:55 azure-windows-latest-4core-16gb TestSyncFullFileSync
2:55 aws linux-2core-8gb TestFilerWorkspaceNotebook/rNb.r
2:46 azure linux-2core-8gb TestFilerReadWrite/workspace_files
2:46 gcp linux-2core-8gb TestFilerWorkspaceFilesExtensionsRead
2:44 azure linux-2core-8gb TestImportDirWithOverwriteFlag
2:37 aws-windows-latest-4core-16gb TestFilerRecursiveDelete/workspace_files
2:34 aws-windows-latest-4core-16gb TestFilerReadWrite/workspace_files_extensions
2:31 azure-windows-latest-4core-16gb TestFilerRecursiveDelete/workspace_files_extensions
2:22 aws-windows-latest-4core-16gb TestFilerWorkspaceFilesExtensionsRead
2:15 azure linux-2core-8gb TestWorkspaceFilesExtensions_ExportFormatIsPreserved/jupyter_scala
2:13 azure linux-2core-8gb TestSyncFullFileSync
2:09 aws linux-2core-8gb TestImportDir
2:06 azure-windows-latest-4core-16gb TestFilerReadWrite/workspace_files_extensions
2:05 gcp linux-2core-8gb TestFilerWorkspaceFilesExtensionsReadDir
2:02 azure-windows-latest-4core-16gb TestLock
2:02 azure-windows-latest-4core-16gb TestImportDirWithOverwriteFlag
2:01 aws linux-2core-8gb TestFilerWorkspaceFilesExtensionsDelete

@yolocs
yolocs force-pushed the stack/docker-shared-workspace-profiles branch from 71222b5 to 79702a7 Compare November 5, 2026 22:12
@yolocs
yolocs marked this pull request as ready for review November 5, 2026 22:13
@yolocs
yolocs requested review from a team as code owners November 5, 2026 22:13

@pietern pietern left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This can have unintended side effects. Let's say you configure Docker assuming profile B, but profile A is earlier in the .databrickscfg for the same workspace host. What happens? Does a docker push use profile A?

I recommend using the "default profile" switcher (run databricks auth switch to see what happens) to be the tie-breaker, if it exists, and keep the error otherwise.

@yolocs

yolocs commented Nov 9, 2026

Copy link
Copy Markdown
Contributor Author

This can have unintended side effects. Let's say you configure Docker assuming profile B, but profile A is earlier in the .databrickscfg for the same workspace host. What happens? Does a docker push use profile A?

I recommend using the "default profile" switcher (run databricks auth switch to see what happens) to be the tie-breaker, if it exists, and keep the error otherwise.

@pietern Yes, it will indeed have potential unintended effects. The problem originates from the credential helper protocol which can only use hostname (not profile) to pick the profile to get the auth token. And our user feedback suggested they'd prefer allowing profiles linking to the same workspace.

Your suggest would only work if one of such profiles is the default profile, right?

@rugpanov
rugpanov removed their request for review November 11, 2026 19:59
@yolocs
yolocs force-pushed the stack/docker-shared-workspace-profiles branch 2 times, most recently from 49a2dc9 to 4b40835 Compare November 12, 2026 22:05
@yolocs yolocs changed the title Allow auth docker configure when profiles share a workspace Record the profile auth docker configure selects for Docker Nov 12, 2026
auth docker configure no longer fails when several profiles match the workspace. It marks the selected profile with docker_credential_helper = true in .databrickscfg, clears the mark from the workspace's other profiles, and warns when the selection switches. auth docker token uses the recorded profile when several match, and auth docker host reports the helper as configured only if Docker's config selects it and it uses that profile. Saving a profile with a different host or workspace ID drops its mark, so it cannot follow the profile to another workspace.

Co-authored-by: Isaac <no-reply@databricks.com>
@yolocs
yolocs force-pushed the stack/docker-shared-workspace-profiles branch from 4b40835 to 68233ed Compare November 12, 2026 22:46
@yolocs

yolocs commented Nov 12, 2026

Copy link
Copy Markdown
Contributor Author

@pietern I'm now trying a better way to disambiguate the profile. It saves a new value in .databrickscfg. PTAL.

@yolocs
yolocs requested a review from pietern November 12, 2026 23:43

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants