Skip to content

air: Report pending compute status in air get JSON - #7455

Merged
pardis-beikzadeh-db merged 4 commits into
mainfrom
air/get-json-pending-status
Nov 11, 2026
Merged

pardis-beikzadeh-db merged 4 commits into
mainfrom
air/get-json-pending-status

Conversation

@pardis-beikzadeh-db

@pardis-beikzadeh-db pardis-beikzadeh-db commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Changes

  • air get -o json now reports PENDING while the run's task is still waiting for compute, matching air list -o json (#6717).
  • Add status_message to air get -o json: the run's current progress message, as the text view shows it in Status. Only set while the run is unfinished.

Why

The Jobs API reports the parent run as RUNNING before its task gets compute, so agents polling air get -o json treated a queued run as started. Internal ticket: CNXT-2498.

Tests

  • New acceptance test acceptance/air/get-pending; acceptance/air/get-ai-runtime now also checks that a finished run has no status_message.
  • go test ./cmd/air, go test ./acceptance -run TestAccept/air/, ./task checks, ./task lint.

Manual test

air get <run_id> -o json on a 1xA10 run while its task waited for compute (status fields only):

w/o fix
"status": "RUNNING",

w/ fix
"status": "PENDING",
"status_message": "Waiting for accelerator compute capacity to become available...",

After cancel, both builds return "status": "CANCELED" with no status_message.

Supersedes #7453 (opened from a fork, so its CI could not run).

This pull request and its description were written by Isaac.

pardis-beikzadeh-db and others added 2 commits November 10, 2026 21:35
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
@github-actions github-actions Bot added the AIR Databricks AI Runtime CLI label Nov 10, 2026
@github-actions

github-actions Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Commit: bd232b8a9 | Workflow run

Regression test report

Selected 2 changed tests (limit=20, 0 not selected)

2 tests: regression=1 unreleased=0 coverage=1 failing=0
test branch main (b00fe63) latest (v1.25.0)
TestAccept/air/get-ai-runtime ✅ ✅ ✅
TestAccept/air/get-pending ✅ ❌ ❌
TestAccept/air/get-pending: branch ✅ | main (b00fe63) ❌ | latest (v1.25.0) ❌

main (b00fe63):

acceptance_test.go:1345: 
        	Error Trace:	/home/runner/work/cli/cli/libs/testdiff/testdiff.go:22
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1345
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1086
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:642
        	Error:      	Not equal: 
        	            	expected: "\n>>> [CLI] air get 123 -o json\n{\n  \"v\": 1,\n  \"ts\": \"[TIMESTAMP]\",\n  \"data\": {\n    \"run_id\": \"123\",\n    \"status\": \"PENDING\",\n    \"started_at\": null,\n    \"duration_seconds\": null,\n    \"attempt_number\": 0,\n    \"experiment_name\": \"my-exp\",\n    \"dashboard_url\": \"[DATABRICKS_URL]/jobs/runs/123?o=[WORKSPACE_ID]\",\n    \"mlflow_url\": null,\n    \"status_message\": \"Waiting for accelerator compute capacity to become available...\",\n    \"est_remaining_seconds\": null,\n    \"est_percent_complete\": null\n  }\n}\n"
        	            	actual  : "\n>>> [CLI] air get 123 -o json\n{\n  \"v\": 1,\n  \"ts\": \"[TIMESTAMP]\",\n  \"data\": {\n    \"run_id\": \"123\",\n    \"status\": \"RUNNING\",\n    \"started_at\": null,\n    \"duration_seconds\": null,\n    \"attempt_number\": 0,\n    \"experiment_name\": \"my-exp\",\n    \"dashboard_url\": \"[DATABRICKS_URL]/jobs/runs/123?o=[WORKSPACE_ID]\",\n    \"mlflow_url\": null,\n    \"est_remaining_seconds\": null,\n    \"est_percent_complete\": null\n  }\n}\n"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -7,3 +7,3 @@
        	            	     "run_id": "123",
        	            	-    "status": "PENDING",
        	            	+    "status": "RUNNING",
        	            	     "started_at": null,
        	            	@@ -14,3 +14,2 @@
        	            	     "mlflow_url": null,
        	            	-    "status_message": "Waiting for accelerator compute capacity to become available...",
        	            	     "est_remaining_seconds": null,
        	Test:       	TestAccept/air/get-pending
        	Messages:   	air/get-pending/output.txt vs /tmp/TestAcceptairget-pending513085872/002/output.txt

latest (v1.25.0): same output as main (b00fe63)

@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Collaborator

✅ Integration test report

Commit: bd232b8

Run: 42919374487

Env 🔄​flaky ✅​pass 🙈​skip Time
✅​ aws linux-2core-8gb 271 17 4:03
✅​ aws-windows-latest-4core-16gb 273 15 3:32
✅​ azure linux-2core-8gb 270 17 3:55
✅​ azure-windows-latest-4core-16gb 272 15 4:41
🔄​ gcp linux-2core-8gb 2 269 17 4:12
✅​ gcp-windows-latest-4core-16gb 273 15 3:21
Test Name gcp linux-2core-8gb
🔄​ TestFsCpFileToDirFileNotOverwritten 🔄​f
🔄​ TestFsCpFileToDirFileNotOverwritten/uc-volumes_to_dbfs 🔄​f
Top 3 slowest tests (at least 2 minutes):
duration env testname
4:34 azure-windows-latest-4core-16gb TestAccept
3:27 aws-windows-latest-4core-16gb TestAccept
3:15 gcp-windows-latest-4core-16gb TestAccept

Co-authored-by: Isaac <no-reply@databricks.com>
@pardis-beikzadeh-db
pardis-beikzadeh-db requested review from caroline-db and removed request for shreyas-goenka November 11, 2026 04:14
@pardis-beikzadeh-db
pardis-beikzadeh-db added this pull request to the merge queue Nov 11, 2026
Merged via the queue into main with commit a0a6430 Nov 11, 2026
28 checks passed
@pardis-beikzadeh-db
pardis-beikzadeh-db deleted the air/get-json-pending-status branch November 11, 2026 23:43
@eng-dev-ecosystem-bot

Copy link
Copy Markdown
Collaborator

❌ Integration test report

Commit: a0a6430

Run: 43122112415

Env ❌​FAIL 🟨​KNOWN 💚​RECOVERED ✅​pass 🙈​skip Time
❌​ aws linux-2core-8gb 48 3 9 1502 1279 223:29
❌​ azure linux-2core-8gb 39 1 11 1491 1287 172:25
❌​ azure-windows-latest-4core-16gb 39 3 9 1410 1314 178:59
❌​ gcp linux-2core-8gb 43 4 8 1494 1285 189:16
❌​ gcp-windows-latest-4core-16gb 43 1 11 1413 1312 189:40
68 interesting tests: 56 FAIL, 6 KNOWN, 6 RECOVERED
Test Name aws linux-2core-8gb azure linux-2core-8gb azure-windows-latest-4core-16gb gcp linux-2core-8gb gcp-windows-latest-4core-16gb
🟨​ TestAccept 🟨​K 🟨​K 🟨​K 🟨​K 🟨​K
❌​ TestAccept/bundle/config-remote-sync/task_rename_revert ✅​p ✅​p 🙈​s ❌​F 🙈​s
❌​ TestAccept/bundle/config-remote-sync/task_rename_revert/DMS=true ✅​p ✅​p ❌​F
❌​ TestAccept/bundle/deployment/bind/dashboard/recreation ❌​F ✅​p ✅​p ✅​p ✅​p
❌​ TestAccept/bundle/deployment/bind/dashboard/recreation/DMS=true/READPLAN= ❌​F ✅​p ✅​p ✅​p ✅​p
❌​ TestAccept/bundle/dms/deployment-metadata-change ✅​p ✅​p ✅​p ✅​p ❌​F
❌​ TestAccept/bundle/dms/deployment-metadata-change/DMS=true ✅​p ✅​p ✅​p ✅​p ❌​F
❌​ TestAccept/bundle/resources/apps/inline_config ✅​p ✅​p ✅​p ❌​F ✅​p
❌​ TestAccept/bundle/resources/apps/inline_config/DMS=true ✅​p ✅​p ✅​p ❌​F ✅​p
🟨​ TestAccept/bundle/resources/apps/lifecycle-started 💚​R 💚​R 💚​R 🟨​K 💚​R
💚​ TestAccept/bundle/resources/apps/lifecycle-started-omitted 💚​R 💚​R 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/apps/lifecycle-started-omitted/DMS=true 💚​R 💚​R 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/apps/lifecycle-started-toggle 💚​R 💚​R 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/apps/lifecycle-started-toggle/DMS= 💚​R 💚​R 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/apps/lifecycle-started-toggle/DMS=true 💚​R 💚​R 💚​R 💚​R 💚​R
🟨​ TestAccept/bundle/resources/apps/lifecycle-started/DMS= 💚​R 💚​R 💚​R 🟨​K 💚​R
🟨​ TestAccept/bundle/resources/apps/lifecycle-started/DMS=true 💚​R 💚​R 💚​R 🟨​K 💚​R
❌​ TestAccept/bundle/resources/clusters/deploy/update-after-create ❌​F ✅​p ✅​p ✅​p ✅​p
❌​ TestAccept/bundle/resources/clusters/deploy/update-after-create/DMS=true ❌​F ✅​p ✅​p ✅​p ✅​p
❌​ TestAccept/bundle/resources/clusters/lifecycle-started-toggle ❌​F ✅​p ✅​p ✅​p ✅​p
❌​ TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS= ❌​F ✅​p ✅​p ✅​p ✅​p
❌​ TestAccept/bundle/resources/clusters/resize-terminated-fallback ❌​F ✅​p ✅​p ✅​p ✅​p
❌​ TestAccept/bundle/resources/clusters/resize-terminated-fallback/DMS= ❌​F ✅​p ✅​p ✅​p ✅​p
❌​ TestAccept/bundle/resources/clusters/resize-terminated-fallback/DMS=true ❌​F ✅​p ✅​p ✅​p ✅​p
❌​ TestAccept/bundle/resources/postgres_branches/update_protected ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/resources/postgres_branches/update_protected/DMS= ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/resources/postgres_branches/update_protected/DMS=true ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/resources/postgres_branches/update_ttl ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/resources/postgres_branches/update_ttl/DMS= ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/resources/postgres_branches/update_ttl/DMS=true ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/resources/postgres_databases/update ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/resources/postgres_databases/update/DMS= ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/resources/postgres_databases/update/DMS=true ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/resources/postgres_endpoints/remove_suspend_timeout ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/resources/postgres_endpoints/remove_suspend_timeout/DMS= ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/resources/postgres_endpoints/remove_suspend_timeout/DMS=true ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/resources/postgres_endpoints/update_autoscaling ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/resources/postgres_endpoints/update_autoscaling/DMS= ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/resources/postgres_endpoints/update_autoscaling/DMS=true ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/resources/postgres_endpoints/update_pg_settings ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/resources/postgres_endpoints/update_pg_settings/DMS= ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/resources/postgres_endpoints/update_pg_settings/DMS=true ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/resources/postgres_endpoints/update_suspend_timeout ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/resources/postgres_endpoints/update_suspend_timeout/DMS= ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/resources/postgres_endpoints/update_suspend_timeout/DMS=true ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/resources/postgres_projects/add_default_endpoint_settings ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/resources/postgres_projects/add_default_endpoint_settings/DMS= ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/resources/postgres_projects/add_default_endpoint_settings/DMS=true ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/resources/postgres_projects/remove_history_retention ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/resources/postgres_projects/remove_history_retention/DMS= ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/resources/postgres_projects/remove_history_retention/DMS=true ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/resources/postgres_projects/update_default_endpoint_suspend ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/resources/postgres_projects/update_default_endpoint_suspend/DMS= ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/resources/postgres_projects/update_default_endpoint_suspend/DMS=true ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/resources/postgres_projects/update_display_name ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/resources/postgres_projects/update_display_name/DMS= ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/resources/postgres_projects/update_display_name/DMS=true ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/resources/postgres_projects/update_history_retention ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/resources/postgres_projects/update_history_retention/DMS= ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/resources/postgres_projects/update_history_retention/DMS=true ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/resources/postgres_roles/update ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/resources/postgres_roles/update/DMS= ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/resources/postgres_roles/update/DMS=true ❌​F ❌​F ❌​F ❌​F ❌​F
🟨​ TestAccept/bundle/resources/postgres_synced_tables/recreate 🟨​K 💚​R 🟨​K 💚​R 💚​R
🟨​ TestAccept/bundle/resources/postgres_synced_tables/recreate/DMS= 🟨​K 💚​R 🟨​K 💚​R 💚​R
💚​ TestAccept/bundle/resources/postgres_synced_tables/recreate/DMS=true 💚​R 💚​R 💚​R 💚​R 💚​R
❌​ TestAccept/bundle/resources/registered_models/basic ✅​p ✅​p ✅​p ✅​p ❌​F
❌​ TestAccept/bundle/resources/registered_models/basic/DMS= ✅​p ✅​p ✅​p ✅​p ❌​F
Top 50 slowest tests (at least 2 minutes):
duration env testname
38:07 aws linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=
34:20 aws linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=true
19:02 aws linux-2core-8gb TestAccept/bundle/resources/clusters/deploy/update-after-create/DMS=
17:24 aws linux-2core-8gb TestAccept/bundle/resources/clusters/deploy/simple/DMS=true
16:45 aws linux-2core-8gb TestAccept/bundle/resources/clusters/deploy/data_security_mode/DMS=true
14:47 aws linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=true
10:02 aws linux-2core-8gb TestAccept/bundle/resources/clusters/deploy/simple/DMS=
9:10 aws linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=true
9:07 aws linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=
8:42 azure-windows-latest-4core-16gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=true
8:38 azure linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=true
8:14 azure linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=
8:09 azure-windows-latest-4core-16gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=
8:01 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=
7:49 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=true
7:26 gcp linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=true
7:22 gcp linux-2core-8gb TestAccept/bundle/resources/clusters/deploy/local_ssd_count/DMS=
7:21 gcp-windows-latest-4core-16gb TestAccept/bundle/invariant/no_drift/DMS=true/INPUT_CONFIG=cluster_apply_policy_default_values.yml.tmpl/READPLAN=
7:16 gcp linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=
7:03 aws linux-2core-8gb TestAccept/bundle/config-remote-sync/multiple_resources/DMS=
6:59 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/deploy/local_ssd_count/DMS=true
6:57 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/deploy/local_ssd_count/DMS=
6:53 gcp linux-2core-8gb TestAccept/bundle/resources/clusters/deploy/local_ssd_count/DMS=true
6:47 azure linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=
6:43 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=true
6:34 gcp linux-2core-8gb TestAccept/bundle/config-remote-sync/multiple_resources/DMS=
6:32 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=
6:24 azure-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=
6:23 azure-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=true
6:14 aws linux-2core-8gb TestAccept/bundle/config-remote-sync/multiple_resources/DMS=true
6:14 gcp linux-2core-8gb TestAccept/bundle/config-remote-sync/multiple_resources/DMS=true
6:02 azure linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=true
5:47 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=
5:35 azure linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=true
5:15 azure linux-2core-8gb TestAccept/bundle/config-remote-sync/multiple_resources/DMS=true
5:13 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=true
5:03 gcp linux-2core-8gb TestAccept/bundle/resources/clusters/deploy/update-after-create/DMS=true
4:59 azure linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=
4:59 gcp linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=true
4:38 gcp linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=
4:36 gcp linux-2core-8gb TestAccept/bundle/invariant/delete_idempotent/DMS=/INPUT_CONFIG=cluster.yml.tmpl/READPLAN=
4:22 gcp-windows-latest-4core-16gb TestAccept/bundle/deploy/spark-jar-task/DMS=
4:20 azure-windows-latest-4core-16gb TestAccept/bundle/deploy/spark-jar-task/DMS=true
3:46 azure-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=
3:46 aws linux-2core-8gb TestAccept/bundle/invariant/destroy_idempotent/DMS=/INPUT_CONFIG=cluster_apply_policy_default_values.yml.tmpl/READPLAN=
3:45 aws linux-2core-8gb TestAccept/bundle/resources/clusters/deploy/data_security_mode/DMS=
3:45 azure-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=true
3:44 gcp linux-2core-8gb TestAccept/bundle/invariant/delete_idempotent/DMS=/INPUT_CONFIG=cluster.yml.tmpl/READPLAN=1
3:42 aws linux-2core-8gb TestAccept/bundle/invariant/no_drift/DMS=true/INPUT_CONFIG=cluster_libraries.yml.tmpl/READPLAN=1
3:41 gcp linux-2core-8gb TestAccept/bundle/invariant/delete_idempotent/DMS=/INPUT_CONFIG=cluster_libraries.yml.tmpl/READPLAN=

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

AIR Databricks AI Runtime CLI

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants