Skip to content

Add a hint when an app's service principal is missing during bundle deploy - #7533

Merged
denik merged 4 commits into
mainfrom
denik/app-missing-sp-hint
Nov 12, 2026
Merged

denik merged 4 commits into
mainfrom
denik/app-missing-sp-hint

Conversation

@denik

@denik denik commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Changes

When bundle deploy updates or starts an app whose service principal is gone, the Apps backend returns a 400 that says to delete the app. The CLI now appends an actionable hint to that error: delete the app with databricks apps delete <name>, wait for the deletion to finish, and deploy again to recreate it.

The hint is added once at the top of the direct engine's app DoUpdate (covering the update call, a non-SUCCEEDED update status, and the start/deploy lifecycle step) and WaitAfterCreate. It matches the substring service principal does not exist or is deleted without regard to case, which is present in both backend variants (the update/start 400 and the compute ERROR status message). The original error and its API details are preserved (wrapped with %w); nothing else changes.

Why

These failures come from a backend issue, not the CLI. Once an app hits this state, every later update or start on it fails until the app is recreated, so a retry would not help. The hint tells users what to do. It does not fix the flaky cloud tests that hit this backend issue; those are already listed in known_failures.

Judgment calls:

  • The hint recommends databricks apps delete and a redeploy rather than bundle destroy, because destroy removes every resource in the bundle.
  • The error is matched on its message text. AGENTS.md says not to branch on error strings, but the backend sends this as a generic 400 BAD_REQUEST or as a compute ERROR status message, so there is no error code or sentinel to match. If the wording changes, the only effect is that the hint stops appearing.
  • The hint is wrapped into the error, so the CLI's (400 BAD_REQUEST) suffix appears after the hint text. Changing the error renderer would be a separate change.
  • Scope is bundle deploy (the direct engine), as requested. bundle run and databricks apps start are not changed.
  • The new acceptance test does not cover the delete-and-redeploy step: the testserver keeps a deleted app in DELETING forever, so the redeploy would only show "unchanged". The direct planner already plans a Create when the app is missing remotely.

Tests

New local acceptance test bundle/resources/apps/missing-service-principal: the testserver returns the backend's 400 for POST /api/2.0/apps/{name}/update and /start. The test checks that the hint appears in both paths. A unit test covers the matcher: the API error, a wrapped error, and the compute ERROR status variant.

Dashboard

  • error/648 | Error: cannot update resources.apps.[NAME]: updating id=[UNIQUE_NAME]: App: [UNIQUE_NAME]'s service principal does not exist or is deleted. App cannot run without a valid service principal. Please delete your app. (400 BAD_REQUEST)

Related, same backend symptom (hint applies too):

  • error/252 | Error: cannot create resources.apps.[NAME]: waiting after creating id=[UNIQUE_NAME]: App: [UNIQUE_NAME]'s service principal does not exist or is deleted. ...
  • error/3340 | Error: cannot update resources.apps.[NAME]: updating id=[UNIQUE_NAME]: failed to reach ACTIVE, got ERROR: App service principal does not exist or is deleted, application cannot run without a valid service principal. Please delete the app and re-create it.

Written by an AI agent (harness_auto, opus5.5, isaac_claude).

This pull request and its description were written by Isaac.

…eploy

Co-authored-by: Isaac <no-reply@databricks.com>
@denik denik added the ai-auto PRs made by AI loops & factories, label Nov 12, 2026
@github-actions github-actions Bot added the DABs DABs related issues label Nov 12, 2026
denik and others added 2 commits November 12, 2026 09:45
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Commit: bea51a5a4 | Workflow run

Regression test report

Selected 1 changed tests (limit=20, 0 not selected)

2 tests: regression=2 unreleased=0 coverage=0 failing=0
test branch main (634e6bf) latest (v1.26.0)
TestAccept/bundle/resources/apps/missing-service-principal/DMS= ✅ ❌ ❌
TestAccept/bundle/resources/apps/missing-service-principal/DMS=true ✅ ❌ ❌
TestAccept/bundle/resources/apps/missing-service-principal/DMS=: branch ✅ | main (634e6bf) ❌ | latest (v1.26.0) ❌

main (634e6bf):

acceptance_test.go:1373: Diff:
        --- bundle/resources/apps/missing-service-principal/output.txt
        +++ /tmp/TestAcceptbundleresourcesappsmissing-service-principalDMS=940786239/002/output.txt
        @@ -11,10 +11,8 @@
         
         >>> errcode [CLI] bundle deploy
         Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files...
        -Error: cannot update resources.apps.mykey: updating id=myappname: App: myappname's service principal does not exist or is deleted. App cannot run without a valid service principal. Please delete your app.
        +Error: cannot update resources.apps.mykey: updating id=myappname: App: myappname's service principal does not exist or is deleted. App cannot run without a valid service principal. Please delete your app. (400 BAD_REQUEST)
         
        -The app cannot be updated or started without its service principal. To recreate the app, delete it with "databricks apps delete myappname", wait for the deletion to finish, and run "databricks bundle deploy" again, using the same profile and target as for this deploy. Deleting the app also deletes its deployment history (400 BAD_REQUEST)
        -
         Endpoint: POST [DATABRICKS_URL]/api/2.0/apps/myappname/update
         HTTP Status: 400 Bad Request
         API error_code: BAD_REQUEST
        @@ -56,10 +53,8 @@
         
         >>> errcode [CLI] bundle deploy
         Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files...
        -Error: cannot update resources.apps.mykey: updating id=myappname: App: myappname's service principal does not exist or is deleted. App cannot run without a valid service principal. Please delete your app.
        +Error: cannot update resources.apps.mykey: updating id=myappname: App: myappname's service principal does not exist or is deleted. App cannot run without a valid service principal. Please delete your app. (400 BAD_REQUEST)
         
        -The app cannot be updated or started without its service principal. To recreate the app, delete it with "databricks apps delete myappname", wait for the deletion to finish, and run "databricks bundle deploy" again, using the same profile and target as for this deploy. Deleting the app also deletes its deployment history (400 BAD_REQUEST)
        -
         Endpoint: POST [DATABRICKS_URL]/api/2.0/apps/myappname/start
         HTTP Status: 400 Bad Request
         API error_code: BAD_REQUEST
        
FAIL

latest (v1.26.0):

acceptance_test.go:1373: Diff:
        --- bundle/resources/apps/missing-service-principal/output.txt
        +++ /tmp/TestAcceptbundleresourcesappsmissing-service-principalDMS=4272411580/002/output.txt
        @@ -11,10 +11,8 @@
         
         >>> errcode [CLI] bundle deploy
         Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files...
        -Error: cannot update resources.apps.mykey: updating id=myappname: App: myappname's service principal does not exist or is deleted. App cannot run without a valid service principal. Please delete your app.
        +Error: cannot update resources.apps.mykey: updating id=myappname: App: myappname's service principal does not exist or is deleted. App cannot run without a valid service principal. Please delete your app. (400 BAD_REQUEST)
         
        -The app cannot be updated or started without its service principal. To recreate the app, delete it with "databricks apps delete myappname", wait for the deletion to finish, and run "databricks bundle deploy" again, using the same profile and target as for this deploy. Deleting the app also deletes its deployment history (400 BAD_REQUEST)
        -
         Endpoint: POST [DATABRICKS_URL]/api/2.0/apps/myappname/update
         HTTP Status: 400 Bad Request
         API error_code: BAD_REQUEST
        @@ -56,10 +53,8 @@
         
         >>> errcode [CLI] bundle deploy
         Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files...
        -Error: cannot update resources.apps.mykey: updating id=myappname: App: myappname's service principal does not exist or is deleted. App cannot run without a valid service principal. Please delete your app.
        +Error: cannot update resources.apps.mykey: updating id=myappname: App: myappname's service principal does not exist or is deleted. App cannot run without a valid service principal. Please delete your app. (400 BAD_REQUEST)
         
        -The app cannot be updated or started without its service principal. To recreate the app, delete it with "databricks apps delete myappname", wait for the deletion to finish, and run "databricks bundle deploy" again, using the same profile and target as for this deploy. Deleting the app also deletes its deployment history (400 BAD_REQUEST)
        -
         Endpoint: POST [DATABRICKS_URL]/api/2.0/apps/myappname/start
         HTTP Status: 400 Bad Request
         API error_code: BAD_REQUEST
TestAccept/bundle/resources/apps/missing-service-principal/DMS=true: branch ✅ | main (634e6bf) ❌ | latest (v1.26.0) ❌

main (634e6bf):

acceptance_test.go:1373: Diff:
        --- bundle/resources/apps/missing-service-principal/output.txt
        +++ /tmp/TestAcceptbundleresourcesappsmissing-service-principalDMS=t1295539975/002/output.txt
        @@ -11,10 +11,8 @@
         
         >>> errcode [CLI] bundle deploy
         Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files...
        -Error: cannot update resources.apps.mykey: updating id=myappname: App: myappname's service principal does not exist or is deleted. App cannot run without a valid service principal. Please delete your app.
        +Error: cannot update resources.apps.mykey: updating id=myappname: App: myappname's service principal does not exist or is deleted. App cannot run without a valid service principal. Please delete your app. (400 BAD_REQUEST)
         
        -The app cannot be updated or started without its service principal. To recreate the app, delete it with "databricks apps delete myappname", wait for the deletion to finish, and run "databricks bundle deploy" again, using the same profile and target as for this deploy. Deleting the app also deletes its deployment history (400 BAD_REQUEST)
        -
         Endpoint: POST [DATABRICKS_URL]/api/2.0/apps/myappname/update
         HTTP Status: 400 Bad Request
         API error_code: BAD_REQUEST
        @@ -56,10 +53,8 @@
         
         >>> errcode [CLI] bundle deploy
         Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files...
        -Error: cannot update resources.apps.mykey: updating id=myappname: App: myappname's service principal does not exist or is deleted. App cannot run without a valid service principal. Please delete your app.
        +Error: cannot update resources.apps.mykey: updating id=myappname: App: myappname's service principal does not exist or is deleted. App cannot run without a valid service principal. Please delete your app. (400 BAD_REQUEST)
         
        -The app cannot be updated or started without its service principal. To recreate the app, delete it with "databricks apps delete myappname", wait for the deletion to finish, and run "databricks bundle deploy" again, using the same profile and target as for this deploy. Deleting the app also deletes its deployment history (400 BAD_REQUEST)
        -
         Endpoint: POST [DATABRICKS_URL]/api/2.0/apps/myappname/start
         HTTP Status: 400 Bad Request
         API error_code: BAD_REQUEST

latest (v1.26.0):

acceptance_test.go:1373: Diff:
        --- bundle/resources/apps/missing-service-principal/output.txt
        +++ /tmp/TestAcceptbundleresourcesappsmissing-service-principalDMS=t3049338114/002/output.txt
        @@ -11,10 +11,8 @@
         
         >>> errcode [CLI] bundle deploy
         Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files...
        -Error: cannot update resources.apps.mykey: updating id=myappname: App: myappname's service principal does not exist or is deleted. App cannot run without a valid service principal. Please delete your app.
        +Error: cannot update resources.apps.mykey: updating id=myappname: App: myappname's service principal does not exist or is deleted. App cannot run without a valid service principal. Please delete your app. (400 BAD_REQUEST)
         
        -The app cannot be updated or started without its service principal. To recreate the app, delete it with "databricks apps delete myappname", wait for the deletion to finish, and run "databricks bundle deploy" again, using the same profile and target as for this deploy. Deleting the app also deletes its deployment history (400 BAD_REQUEST)
        -
         Endpoint: POST [DATABRICKS_URL]/api/2.0/apps/myappname/update
         HTTP Status: 400 Bad Request
         API error_code: BAD_REQUEST
        @@ -56,10 +53,8 @@
         
         >>> errcode [CLI] bundle deploy
         Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files...
        -Error: cannot update resources.apps.mykey: updating id=myappname: App: myappname's service principal does not exist or is deleted. App cannot run without a valid service principal. Please delete your app.
        +Error: cannot update resources.apps.mykey: updating id=myappname: App: myappname's service principal does not exist or is deleted. App cannot run without a valid service principal. Please delete your app. (400 BAD_REQUEST)
         
        -The app cannot be updated or started without its service principal. To recreate the app, delete it with "databricks apps delete myappname", wait for the deletion to finish, and run "databricks bundle deploy" again, using the same profile and target as for this deploy. Deleting the app also deletes its deployment history (400 BAD_REQUEST)
        -
         Endpoint: POST [DATABRICKS_URL]/api/2.0/apps/myappname/start
         HTTP Status: 400 Bad Request
         API error_code: BAD_REQUEST
        
FAIL

@denik
denik requested a review from andrewnester November 12, 2026 10:04
@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Collaborator

✅ Integration test report

Commit: bea51a5

Run: 43199404531

Env 🔄​flaky 💚​RECOVERED ✅​pass 🙈​skip Time
💚​ aws linux-2core-8gb 1 270 16 9:36
💚​ aws-windows-latest-4core-16gb 1 272 14 12:00
🔄​ azure linux-2core-8gb 2 1 267 16 8:37
💚​ azure-windows-latest-4core-16gb 1 271 14 7:53
💚​ gcp linux-2core-8gb 1 270 16 13:34
💚​ gcp-windows-latest-4core-16gb 1 272 14 14:40
Test Name aws linux-2core-8gb aws-windows-latest-4core-16gb azure linux-2core-8gb azure-windows-latest-4core-16gb gcp linux-2core-8gb gcp-windows-latest-4core-16gb
💚​ TestAccept 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
🔄​ TestFsCpFileToDirWithOverwriteFlag ✅​p ✅​p 🔄​f ✅​p ✅​p ✅​p
🔄​ TestFsCpFileToDirWithOverwriteFlag/uc-volumes_to_uc-volumes ✅​p ✅​p 🔄​f ✅​p ✅​p ✅​p
Top 13 slowest tests (at least 2 minutes):
duration env testname
4:02 azure-windows-latest-4core-16gb TestAccept
3:51 azure linux-2core-8gb TestAccept
3:49 aws-windows-latest-4core-16gb TestAccept
3:48 aws linux-2core-8gb TestAccept
3:33 gcp linux-2core-8gb TestAccept
3:33 gcp-windows-latest-4core-16gb TestAccept
2:55 gcp-windows-latest-4core-16gb TestFsCpDirToDirFileNotOverwritten/dbfs_to_uc-volumes
2:37 gcp linux-2core-8gb TestLock
2:26 gcp-windows-latest-4core-16gb TestFsCpDirToDirWithOverwriteFlag/local_to_dbfs
2:09 aws-windows-latest-4core-16gb TestFsCpDir/dbfs_to_uc-volumes
2:09 gcp-windows-latest-4core-16gb TestFsCpDirToDirWithOverwriteFlag/dbfs_to_dbfs
2:01 gcp-windows-latest-4core-16gb TestFsCpFileToFileWithOverwriteFlag/dbfs_to_dbfs
2:00 gcp-windows-latest-4core-16gb TestFsCpDir/dbfs_to_local

aws: TestAccept/bundle/resources/apps/missing-service-principal

Co-authored-by: Isaac <no-reply@databricks.com>
@denik
denik marked this pull request as ready for review November 12, 2026 10:49
@denik
denik requested review from a team as code owners November 12, 2026 10:49
@denik
denik enabled auto-merge November 12, 2026 10:49
return err
}
return fmt.Errorf("%w\n\nThe app cannot be updated or started without its service principal. "+
"To recreate the app, delete it with \"databricks apps delete %s\", wait for the deletion to finish, "+

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

DoCreate already retries while the app is DELETING (app.go:143) and IsGone plans a create for it, so "wait for the deletion to finish" isn't needed. Drop that clause?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the clause is still needed. IsGone is only checked for resources being deleted (bundle_plan.go, delete branch, and deleteConfirmedGone in apply.go). For an app still in config, the planner reads the app, and while it is DELETING, GET still returns it. A redeploy then plans an update, not a create, and the DoCreate DELETING retry is never reached. I checked on the testserver: bundle deploy, then apps delete myappname, then bundle plan shows update apps.mykey (Plan: 0 to add, 1 to change). A create is planned only after GET returns 404, which means after the deletion finishes. So I left the hint as is in the follow-up #7545. Happy to drop the clause if you would rather handle DELETING in the planner instead.

AI agent (harness_auto, opus5.5, isaac_claude)

@@ -0,0 +1,13 @@
trace $CLI bundle deploy
rm -f out.requests.txt

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This rm -f is a no-op: the harness records to $OUT_REQUESTS in a sibling dir (acceptance_test.go:913), so the first print_requests also lists the create. Drop the line?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, thanks. It is a no-op; removed in #7545 (the golden output is unchanged, since the first print_requests already listed the create).

AI agent (harness_auto, opus5.5, isaac_claude)

@denik
denik added this pull request to the merge queue Nov 12, 2026
Merged via the queue into main with commit 4668f26 Nov 12, 2026
28 checks passed
@denik
denik deleted the denik/app-missing-sp-hint branch November 12, 2026 11:20
tedjt pushed a commit to tedjt/cli that referenced this pull request Nov 12, 2026
## Changes

Follow-up to [databricks#7533](databricks#7533),
addressing review feedback: remove `rm -f out.requests.txt` from the
`bundle/resources/apps/missing-service-principal` script. The harness
records requests to `$OUT_REQUESTS` in a separate output dir, so the
line did nothing. The golden output is unchanged: the first
`print_requests.py` already listed the create request.

The hint text from databricks#7533 stays as is ("wait for the deletion to
finish"). Right after `databricks apps delete`, the app is still
returned in DELETING, and `bundle plan` plans an update, not a create.
`IsGone` is only checked for resources being deleted. So a redeploy run
before the deletion finishes does not recreate the app. Details are in
the review thread on databricks#7533.

## Tests

`go test ./acceptance -run
TestAccept/bundle/resources/apps/missing-service-principal` passes
locally with no output change.

_Written by an AI agent (harness_auto, opus5.5, isaac_claude)._

This pull request and its description were written by Isaac.

Co-authored-by: Isaac <no-reply@databricks.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ai-auto PRs made by AI loops & factories, DABs DABs related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants