Repository navigation
Add data_quality_monitors bundle resource (direct engine) - #7094
Conversation
Adds the data-quality/v1 Monitor API as a direct-engine bundle resource, alongside the legacy quality_monitors resource it supersedes. Co-authored-by: Isaac <no-reply@databricks.com>
Integration test reportCommit: 5a32e8e
Top 14 slowest tests (at least 2 minutes):
|
…ceptance test Co-authored-by: Isaac <no-reply@databricks.com>
|
Commit: Regression test reportSelected 13 changed tests (limit=20, 0 not selected) 658 tests: regression=14 unreleased=0 coverage=644 failing=0
14 more failure details omitted, see the full report in the run. |
…y monitors Co-authored-by: Isaac <no-reply@databricks.com>
…n alert Co-authored-by: Isaac <no-reply@databricks.com>
There was a problem hiding this comment.
what's the purpose of this folder vs resources/data_quality_monitors/*? let's remove/fold it into the others if this test offers additional coverage
There was a problem hiding this comment.
Folded it into resources/data_quality_monitors/data_profiling: it now has a dev target and the script starts by validating it (schedule dropped), while the default target's deploy keeps it. Removed the separate folder.
| # Create:REQ Update:IGN in the data-quality API: the update endpoint ignores it. | ||
| # Same as quality_monitors.assets_dir. | ||
| - field: data_profiling_config.assets_dir | ||
| reason: immutable |
There was a problem hiding this comment.
can this be annotated in the upstream proto to get it out of the box (it'd show up under data_quality_monitors.generated.yml as spec:immutable). Hand-writing it is a smell
There was a problem hiding this comment.
Yes: databricks-eng/universe#2963330 marks assets_dir IMMUTABLE in the OpenAPI spec (spec-only, since the backend's ApiUtil.update rejects * masks touching a google.api.field_behavior IMMUTABLE field). I'll swap the hand-written entry for the generated one once an SDK bump picks it up.
There was a problem hiding this comment.
I'll swap the hand-written entry for the generated one once an SDK bump picks it up.
I'm pretty sure the SDK bump will do this automatically (DABs should be set up to error on the field conflict) so no action from you besides merging that upstream PR :-)
Co-authored-by: Isaac <no-reply@databricks.com>
## Release v1.24.0 ### Notable Changes * A variable used as the value of a secret is now sensitive: referencing it from any other field, such as a job parameter or a secret's `comment`, is an error because the value would be used in plain text. Mark other secret-holding variables with `sensitive: true` to get the same protection. ([#7288](#7288)) ### CLI * Fix `databricks auth login` and `databricks configure` overwriting the parent profile's host and token when saving a profile with a dotted name such as `team.prod` next to an existing `[team]`. ([#7266](#7266)) * `databricks auth login` and `databricks auth token` now reject the name of an existing profile after "Create a new profile" instead of logging in to that profile. ([#7196](#7196)) * `databricks auth login` and `databricks auth token` now reject a host that is not a URL with a hostname, such as `/`, `?o=123` or `https://`, instead of falling back to `DATABRICKS_HOST` or failing later. An empty host after "Enter a host URL manually" in the login profile picker no longer starts discovery login. ([#7195](#7195)) * Prompts that validate their input, such as the `databricks configure` host prompt, no longer open with a red cross before anything is typed. ([#7314](#7314)) * `databricks ssh connect` now documents `--environment-version` for choosing the serverless environment version, starts a separate default-named server for each version, and rejects the flag together with `--cluster`. Default connection names also include the environment version in their hash, so the default name when `--environment-version` is not specified changes with this release. Configuration associated with the old connection name (secret scope, SSH keys and known hosts in your local `.databricks` folder) will no longer be used and can be safely removed. ([#7310](#7310)) ### AI Runtime * Show workspace-supported accelerator types in `databricks air run -h config.compute.accelerator_type`. ([#7275](#7275)) * `air list` now reports `No active runs found.` for an empty active-only result. ([#7276](#7276)) * Stop printing a retry notice on every `air run --watch`; retries are still reported when they happen. ([#7270](#7270)) ### Bundles * Fix `bundle plan` reporting an update to an app's `config.env` values on every run, which also redeployed the app on every `bundle deploy`. The Apps API returns environment variable names only, so `value` and `value_from` are no longer compared against the remote deployment. ([#7326](#7326)) * Add the `data_quality_monitors` resource for data quality monitors on Unity Catalog schemas and tables (direct engine only). ([#7094](#7094)) * `bundle deploy` now asks for confirmation before recreating or deleting a Feature View, and `bundle deploy` and `bundle destroy` warn that FeatureSpecs using a Feature View are not checked. ([#7164](#7164)) * Accept grant `privileges` values that the SDK enum does not list yet: `bundle validate` no longer warns (or errors under `--strict`) on them, and the PyDABs `Privilege` enum accepts unknown strings instead of raising. The JSON schema and PyDABs still advertise the known values for completion. ([#7315](#7315)) * Add the `materialization_sets` resource, which materializes Feature Store Feature Views to offline and online stores. ([#7164](#7164)) * Detect notebooks with the same rule as the workspace: a `.py`, `.sql`, `.scala` or `.r` file is a notebook when its first line contains `Databricks notebook source`, for example after a byte order mark. Previously such a file was tracked as a plain file, so removing it locally left the remote notebook behind. ([#7290](#7290)) * Add `sensitive: true` for bundle variables. A variable used as the value of a secret is sensitive by default. `bundle validate -o json` redacts only sensitive variables instead of every string variable, and a sensitive variable can only be referenced from a sensitive resource field (such as a secret's `value`) or from another sensitive variable. ([#7288](#7288)) ### Dependency Updates * Bump dependencies with known vulnerabilities. ([#7305](#7305)) * Bump dependencies with known vulnerabilities. ([#7327](#7327)) * Update the Go SDK to [v0.191.1](https://github.com/databricks/databricks-sdk-go/releases/tag/v0.191.1). ([#7277](#7277)) * Update the Go SDK to [v0.193.0](https://github.com/databricks/databricks-sdk-go/releases/tag/v0.193.0). ([#7303](#7303))
## Changes Add `python/docs/.gitattributes` marking `index.rst` and the per-resource `databricks.bundles.*.rst` pages as `linguist-generated`. `databricks.bundles.core.rst` is hand-written and stays visible. ## Why pydabs-codegen has written these pages since databricks#6727 (`python/codegen/codegen/generated_docs.py`), but the gitattributes from databricks#6394 predate it and only cover the Python modules and test data. So new-resource PRs like databricks#7094 show the doc pages as regular diffs. `python/databricks_tests/core/public_api.txt` is left unmarked on purpose: it's the golden snapshot of the hand-written `core` API, and reviewers are meant to see changes to it. ## Tests `git check-attr linguist-generated` resolves to `true` for `index.rst` and `databricks.bundles.jobs.rst`, and to `false` for `databricks.bundles.core.rst`. This pull request and its description were written by Isaac. Co-authored-by: Isaac <no-reply@databricks.com>
…icks#7181) ## Changes The regression report comment now puts the per-test table in a `<details>` block, with the counts line (`659 tests: regression=15 ...`) as its summary. The workflow trims an oversized report by splitting it at `<details>`, so it now always keeps the first block (the table) and drops only failure details. ## Why The table has a row per test variant, often hundreds (e.g. [databricks#7094](databricks#7094 (comment))), which buries the failure details below it. ## Tests - Extended `TestRenderRegressionReport` to assert the table is wrapped. - Ran the workflow's trimming logic in node on a synthetic oversized report (700-row table, 20 failures): the table is kept, 8 failures fit, 12 are reported as omitted, and `<details>` tags stay balanced. This pull request and its description were written by Isaac. Co-authored-by: Isaac <no-reply@databricks.com>
…atabricks#7307) ## Changes Stop selecting an entire invariant suite when its `test.toml` only adds exclusions. Script changes, removed exclusions, and overrides of inherited rules still select the suite. The new Git-backed `acceptance/selftest/selection_exclusions` has two suites, `migrate` and `no_drift`. Adding `new.yml.tmpl` and excluding it from `migrate` selects only `no_drift/INPUT_CONFIG=new.yml.tmpl`. Changing the migration script or removing its exclusion selects the migration suite again. The harness also omits unselected variants from the report and explicitly skips a directory if no selected variants remain after exclusions. ## Why The [report on databricks#7094](databricks#7094 (comment)) had 660 rows for 14 selected entries: 551 filtered-out variants and 84 unrelated migration cases. Those rows filled the comment budget and hid all 16 failure details. `limit=20` caps selection entries; one directory entry can expand to many matrix variants. ## Tests - `./task fmt`, `./task checks`, and `./task lint` pass. - `./task test` passes. Both variants of the existing `presets_name_prefix_dev` test passed on retry after a temporary sandbox `/tmp/.git` artifact caused Git metadata mismatches. - Git-backed acceptance self-test covers adding an exclusion, changing the script, and removing the exclusion. - Targeted selection, regression-report, and subset unit tests pass. - Verified with the actual harness that combination exclusions removing every selected variant produce a skip. Replayed `acceptance/selftest/selection_exclusions/script` against the pre-fix selector (`050b89d`) and this PR. For the first scenario, adding a template and excluding it from migration: Before the fix: ```text Selected 2 changed tests (limit=50, 0 not selected) 10 ^bundle$/^invariant$/^no_drift$/^INPUT_CONFIG=new\.yml\.tmpl$ 5 ^bundle$/^invariant$/^migrate$ ``` With this PR: ```text Selected 1 changed tests (limit=50, 0 not selected) 10 ^bundle$/^invariant$/^no_drift$/^INPUT_CONFIG=new\.yml\.tmpl$ ``` The extra `migrate` entry selected all existing migration variants even though the config edit only added an exclusion. The other two scenarios print identically before and after: changing the migration script or removing the exclusion still selects migration. _This PR and its description were written by Codex._
Changes
Adds a
data_quality_monitorsbundle resource for the data-quality/v1 Monitor API (anomaly detection on schemas, data profiling on tables). Direct engine only. The legacyquality_monitorsresource is unchanged; deprecating it is a separate follow-up.bundle/config/resources/data_quality_monitor.goembedsdataquality.Monitor. The ID is<object_type>,<object_id>, the same as the Terraform import ID fordatabricks_data_quality_monitor.bundle/direct/dresources/data_quality_monitor.go: CRUD onw.DataQuality. Update sendsupdate_mask: *.configs/data_quality_monitors.yml:object_typeandobject_idare ID fields;data_profiling_config.assets_dirrecreates on change (the update API ignores it). Output-only fields and the input-onlyskip_builtin_dashboard/warehouse_id(never returned on GET) come from the generated config.quality_monitors:run_asis rejected (the API has none), and presets that pause triggers dropdata_profiling_config.schedule(itspause_statusis read-only). The monitor has no name, so there is no name prefix. It has no permissions and no workspace URL, becauseobject_idis a UUID.Why
Data quality monitoring is moving to GA on the data-quality/v1 API, and
quality_monitorsonly covers the legacy table-monitor API. This resource lets bundles manage v1 monitors, including schema-level anomaly detection, which the legacy resource can't express.Notes for reviewers:
ListMonitoris unimplemented server-side, so lookup variables aren't supported for this resource.snapshot: {}is lost when a monitor is defined in Python.quality_monitorshas the same pre-existing limitation; it's visible in thedata_quality_monitors-supportgolden.*,anomaly_detection_config.notification_settingsis replaced on every update, so omitting it removes an alert set in the Data Quality Monitoring UI. The bundle is the source of truth; this is documented on the resource.data_quality_monitorsto the deployment metadata service, and databricks-eng/universe#2867273 marked the input-only fields in the spec (in SDK v0.191.0).Tests
all_test.go,state_load_test.go,resources_test.go, and the mutator tests.apply_target_mode_testasserts the dev-mode schedule drop.resources/data_quality_monitors/basic: schema monitor that references${resources.schemas.x.schema_id}. Covers create, in-place update, no drift, recreate on an identity change, and destroy.data_profiling: table monitor. Covers output-only fields not drifting, in-place schedule update, andassets_dirrecreate.remote-delete: an out-of-band delete is detected.lifecycle: cloud-capable.bundle/data_quality_monitor: dev-mode target drops the schedule; production keeps it.deployment/bind/data_quality_monitor: bind and unbind, and a malformed ID is rejected.python/data_quality_monitors-support.golangci-lint run ./...reports 0 issues. tidy, whitespace, deadcode, codeowners and lockfile checks pass, and all generators are idempotent. 505 PyDABs tests pass.lifecyclepasses on all clouds, with and without deployment history (DMS).This pull request and its description were written by Isaac.