Skip to content

acc: don't record quality monitor status and dashboard_id - #7495

Merged
denik merged 2 commits into
mainfrom
fix-qm-status-flaky
Nov 12, 2026
Merged

denik merged 2 commits into
mainfrom
fix-qm-status-flaky

Conversation

@denik

@denik denik commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Changes

Project quality monitor GET responses and plan remote_state through a monitor_fields() allow-list in acceptance/bundle/resources/quality_monitors/script.prepare. This follows .agents/rules/testing.md and matches postgres_*/script.prepare. The allow-list leaves out status and dashboard_id. It applies to create/out.plan_noop.json, change_*/out.plan.json and change_table_name/out.get.json. The dashboard_id repl in test.toml had nothing left to match, so I removed it.

Why

These tests run bundle plan / quality-monitors get right after the monitor is created or recreated. When that happens, the backend sometimes still reports MONITOR_STATUS_PENDING and hasn't set dashboard_id yet. The golden then doesn't match, and the tests are flaky on AWS, Azure and GCP.

Decision: record less instead of polling until the monitor is ACTIVE. Both fields are output-only and change on the server's schedule. None of these tests is about them: they cover plan actions, requests and state, and those are all still recorded. Polling would make the tests slower and would depend on how long provisioning takes on the backend.

Dashboard:

error verbatim snippet
error/1405 "status": "MONITOR_STATUS_PENDING", (expected "status": "MONITOR_STATUS_ACTIVE",, "dashboard_id" missing) in create/out.plan_noop.json and change_table_name/out.get.json
error/2893 --- .../out.plan.json - "dashboard_id": "(redacted)", - "status": "MONITOR_STATUS_ACTIVE", + "status": "MONITOR_STATUS_PENDING",

Tests: TestAccept/bundle/resources/quality_monitors/create/DMS=, TestAccept/bundle/resources/quality_monitors/change_table_name/DMS=, TestAccept/bundle/resources/quality_monitors/change_table_name/DMS=true (also change_assets_dir and change_output_schema_name).

Tests

  • Local: go test ./acceptance -run TestAccept/bundle/resources/quality_monitors passes.
  • Cloud: all quality_monitors subtests pass on AWS (-count=1), both before and after the switch to the allow-list. Azure and GCP credentials were not available locally, so those clouds are left to CI.

This pull request and its description were written by Isaac.

Right after a monitor is created, the backend can still report
MONITOR_STATUS_PENDING without a dashboard_id, which made the
quality_monitors acceptance tests flaky on cloud.

Co-authored-by: Isaac <no-reply@databricks.com>
@denik denik added the ai-auto PRs made by AI loops & factories, label Nov 11, 2026
@github-actions github-actions Bot added DABs DABs related issues test-only-fix PRs that update failing of flaky tests without changing CLI code. labels Nov 11, 2026
@github-actions

github-actions Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Commit: a9e4d5073 | Workflow run

Regression test report

Selected 4 changed tests (limit=20, 0 not selected)

8 tests: regression=0 unreleased=0 coverage=8 failing=0
test branch main (4668f26) latest (v1.26.0)
TestAccept/bundle/resources/quality_monitors/change_assets_dir/DMS= ✅ ✅ ✅
TestAccept/bundle/resources/quality_monitors/change_assets_dir/DMS=true ✅ ✅ ✅
TestAccept/bundle/resources/quality_monitors/change_output_schema_name/DMS= ✅ ✅ ✅
TestAccept/bundle/resources/quality_monitors/change_output_schema_name/DMS=true ✅ ✅ ✅
TestAccept/bundle/resources/quality_monitors/change_table_name/DMS= ✅ ✅ ✅
TestAccept/bundle/resources/quality_monitors/change_table_name/DMS=true ✅ ✅ ✅
TestAccept/bundle/resources/quality_monitors/create/DMS= ✅ ✅ ✅
TestAccept/bundle/resources/quality_monitors/create/DMS=true ✅ ✅ ✅

@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Collaborator

✅ Integration test report

Commit: a9e4d50

Run: 43215830816

Env 🔄​flaky 💚​RECOVERED ✅​pass 🙈​skip Time
💚​ aws linux-2core-8gb 1 282 15 14:16
🔄​ aws-windows-latest-4core-16gb 1 1 283 13 9:18
🔄​ azure linux-2core-8gb 2 1 279 15 14:29
💚​ azure-windows-latest-4core-16gb 1 283 13 9:54
💚​ gcp linux-2core-8gb 1 282 15 11:15
💚​ gcp-windows-latest-4core-16gb 1 284 13 7:04
Test Name aws linux-2core-8gb aws-windows-latest-4core-16gb azure linux-2core-8gb azure-windows-latest-4core-16gb gcp linux-2core-8gb gcp-windows-latest-4core-16gb
💚​ TestAccept 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
🔄​ TestSyncIncrementalFileSync ✅​p 🔄​f ✅​p ✅​p ✅​p ✅​p
🔄​ TestSyncIncrementalSyncFileToPythonNotebook ✅​p ✅​p 🔄​f ✅​p ✅​p ✅​p
🔄​ TestSyncNestedFolderSync ✅​p ✅​p 🔄​f ✅​p ✅​p ✅​p
Top 17 slowest tests (at least 2 minutes):
duration env testname
4:38 azure-windows-latest-4core-16gb TestFilerWorkspaceFilesExtensionsReadDir
3:59 azure linux-2core-8gb TestAccept
3:53 aws linux-2core-8gb TestAccept/bundle/resources/quality_monitors/change_table_name/DMS=
3:41 aws-windows-latest-4core-16gb TestAccept
3:39 azure-windows-latest-4core-16gb TestAccept
3:24 aws linux-2core-8gb TestAccept
3:18 gcp-windows-latest-4core-16gb TestAccept
3:12 azure-windows-latest-4core-16gb TestAccept/bundle/resources/quality_monitors/change_output_schema_name/DMS=
3:11 gcp linux-2core-8gb TestAccept
2:42 aws-windows-latest-4core-16gb TestAccept/bundle/resources/quality_monitors/change_assets_dir/DMS=true
2:30 aws linux-2core-8gb TestFilerWorkspaceFilesExtensionsReadDir
2:25 aws linux-2core-8gb TestAccept/bundle/resources/quality_monitors/change_assets_dir/DMS=true
2:14 aws-windows-latest-4core-16gb TestAccept/bundle/resources/quality_monitors/change_output_schema_name/DMS=
2:11 aws-windows-latest-4core-16gb TestAccept/bundle/resources/quality_monitors/change_output_schema_name/DMS=true
2:05 gcp-windows-latest-4core-16gb TestImportDir
2:02 azure linux-2core-8gb TestAccept/bundle/resources/quality_monitors/change_assets_dir/DMS=true
2:01 azure-windows-latest-4core-16gb TestFilerRecursiveDelete/workspace_files_extensions

@denik
denik marked this pull request as ready for review November 11, 2026 18:16
@denik
denik requested review from a team as code owners November 11, 2026 18:16
@denik
denik enabled auto-merge November 11, 2026 18:16
@andrewnester
andrewnester self-requested a review November 12, 2026 08:58

@andrewnester andrewnester left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The integration report for this commit still marks change_table_name flaky on aws-windows (both variants) and azure-windows DMS=true. Was that attempt the same status/dashboard_id diff, or a different cause?

@denik
denik disabled auto-merge November 12, 2026 12:05
…-list

Replace the jq del() deny-list with an allow-list helper in
quality_monitors/script.prepare, as .agents/rules/testing.md asks, and reuse
it for remote_state in the plan goldens.

Co-authored-by: Isaac <no-reply@databricks.com>
@denik

denik commented Nov 12, 2026

Copy link
Copy Markdown
Contributor Author

AI agent (harness_auto, opus5.5, isaac_claude): Re the review question ("Was that attempt the same status/dashboard_id diff, or a different cause?"): it was a different cause, unrelated to this change. In all three flaky attempts on that run (aws-windows DMS= and DMS=true, azure-windows DMS=true), the first bundle deploy failed while uploading deployment state:

 Created quality_monitors.monitor1
+Error: Post "[DATABRICKS_URL]/api/2.0/workspace-files/import-file/Workspace%2FUsers%2F[USERNAME]%2F.bundle%2Fquality-monitor-update-[UNIQUE_NAME]%2Fdefault%2Fstate%2Fmetadata.json?overwrite=true": request timed out after 1m30s of inactivity

On azure-windows the timed-out file was state/resources.json. That attempt never reached the plan/get steps, so it never compared status or dashboard_id. The retry passed in each case. It looks like a transient workspace-files timeout on Windows runners, and this test only hits it because it sits in that environment. I'm not addressing it in this PR.

@denik
denik requested a review from andrewnester November 12, 2026 12:12
@denik
denik enabled auto-merge November 12, 2026 12:26
@denik
denik added this pull request to the merge queue Nov 12, 2026
Merged via the queue into main with commit 44261c3 Nov 12, 2026
27 checks passed
@denik
denik deleted the fix-qm-status-flaky branch November 12, 2026 12:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ai-auto PRs made by AI loops & factories, DABs DABs related issues test-only-fix PRs that update failing of flaky tests without changing CLI code.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants