Repository navigation
[FS2524] Add Feature Spec as a Bundle Resource - #7364
Merged
Merged
Conversation
Dropped when rebasing onto the bot's Update Go SDK to v0.194.0 PR. Co-authored-by: Isaac <no-reply@databricks.com>
A feature spec has create, get and delete and no update, and every field is immutable, so any change recreates it. It is a UC function, so its grants go through the existing grants sub-resource with securable type "function"; on a recreate the grants node plans an update and re-applies them. The map-like pair lists (rename_outputs, default_values, input_bindings) are compared by key, and exclude_columns and the extra pip requirements are sorted on both sides, so reordering them is not a recreate. features and the lookup keys stay positional: their order sets the output columns and must match the table's primary key. Co-authored-by: Isaac <no-reply@databricks.com>
… FeatureSpec Replaced by the generator after rebasing onto the SDK bump. Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Also give the fake's feature spec map its own block and shorten the recreate prompt. Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Contributor
|
Commit: Regression test reportSelected 14 changed tests (limit=20, 0 not selected) 21 tests: regression=14 unreleased=0 coverage=7 failing=0
TestAccept/bundle/deployment/bind/feature_spec/DMS=: branch ✅ | main (6bdf874) ❌ | latest (v1.25.0) ❌main (6bdf874): latest (v1.25.0): same output as main (6bdf874) TestAccept/bundle/deployment/bind/feature_spec/DMS=true: branch ✅ | main (6bdf874) ❌ | latest (v1.25.0) ❌main (6bdf874): latest (v1.25.0): same output as main (6bdf874) TestAccept/bundle/python/feature_specs-support/DMS=/PYDAB_VERSION=current: branch ✅ | main (6bdf874) ❌ | latest (v1.25.0) ❌main (6bdf874): latest (v1.25.0): same output as main (6bdf874) TestAccept/bundle/python/feature_specs-support/DMS=true/PYDAB_VERSION=current: branch ✅ | main (6bdf874) ❌ | latest (v1.25.0) ❌main (6bdf874): latest (v1.25.0): same output as main (6bdf874) TestAccept/bundle/refschema/DMS=: branch ✅ | main (6bdf874) ❌ | latest (v1.25.0) ❌main (6bdf874): latest (v1.25.0): TestAccept/bundle/refschema/DMS=true: branch ✅ | main (6bdf874) ❌ | latest (v1.25.0) ❌main (6bdf874): latest (v1.25.0): TestAccept/bundle/resource_deps/implicit_deps_feature_spec/DMS=: branch ✅ | main (6bdf874) ❌ | latest (v1.25.0) ❌main (6bdf874): latest (v1.25.0): same output as main (6bdf874) TestAccept/bundle/resource_deps/implicit_deps_feature_spec/DMS=true: branch ✅ | main (6bdf874) ❌ | latest (v1.25.0) ❌main (6bdf874): latest (v1.25.0): same output as main (6bdf874) TestAccept/bundle/resources/feature_specs/basic/DMS=: branch ✅ | main (6bdf874) ❌ | latest (v1.25.0) ❌main (6bdf874): latest (v1.25.0): TestAccept/bundle/resources/feature_specs/order/DMS=: branch ✅ | main (6bdf874) ❌ | latest (v1.25.0) ❌main (6bdf874): latest (v1.25.0): same output as main (6bdf874) 4 more failure details omitted, see the full report in the run. |
Collaborator
✅ Integration test reportCommit: f34d1a8
Top 24 slowest tests (at least 2 minutes):
|
Every feature spec case that deploys now sets Cloud = true, and the
invariant matrix no longer skips feature_spec.yml.tmpl on cloud.
Creating a feature spec resolves its lookup tables and UDFs in Unity
Catalog, so each case creates them in a per-run main.fspec_<unique>
schema with create_feature_spec_fixtures.py and drops the schema on
exit. Bundle names carry $UNIQUE_NAME so parallel runs do not share
state. The grants case grants to account users, which exists in every
workspace. The bind case binds a spec created outside the bundle and
plans no change.
The fake's delete now returns {} like the real endpoint.
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Daniel114510
commented
Nov 5, 2026
Daniel114510
commented
Nov 5, 2026
Daniel114510
commented
Nov 5, 2026
Co-authored-by: Isaac <no-reply@databricks.com>
Daniel114510
changed the title
Regenerated from the FeatureSpec API in .codegen/cli.json (Go SDK v0.194.0): the bundle JSON schema, required-field validation, the direct engine API types, the refschema field list and the PyDABs models. configs/feature_specs.generated.yml regenerates unchanged. Fills the three schema placeholders the resource adds, following the features and registered_models entries. The grants example uses EXECUTE, since a feature spec is a UC function. Adds the python/feature_specs-support golden. Co-authored-by: Isaac <no-reply@databricks.com>
main now builds with Go 1.27, whose modernize embedlit check flags the embedded type name in a struct literal. Co-authored-by: Isaac <no-reply@databricks.com>
&>> needs bash 4; the macOS runners use bash 3.2, which rejects the scripts before they run. Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
UC lowercases a feature spec's name, and create and get now return it that way. Listing name in provided_id_fields skips that read-back difference, as it does for a schema's name, so a mixed-case name no longer recreates the feature spec on every plan. The fake lowercases the name like the real endpoint. Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Daniel114510
enabled auto-merge
November 11, 2026 09:10
Member
There was a problem hiding this comment.
Nits, none blocking:
grants/feature_specs/script:15and:23usejq 'del(.features)';nostampis the helper that strips the plan header, and theregistered_modelsgrants test uses it.messages.go:66and:122say "feature specs" while the sibling messages say "Feature Views".python/feature_specs-support/test.toml:1setsCloud = trueanddatabricks.ymlkeeps agrantsblock;acceptance/bundle/python/README.mdasks forCloud = falseand no cloud-only blocks.feature_spec.go:90says whatslices.Clonedoes to an empty list but not that the function returns a sorted copy.
(Edited: dropped the asks about the cloud run and about restating decisions and API requests in the description. Cloud = true tests run in the PR's Integration Tests check, and the code carries the rest.)
janniklasrose
added a commit
that referenced
this pull request
Nov 11, 2026
…ource skills The skills asked for things a hands-off author cannot usefully supply and a reviewer should not demand: a cloud run link and environment in the description, a restatement of API asks and bundle-wide decisions there, and a shared helper whenever a block repeated a sibling's. Running review-new-resource on #7364 produced exactly those comments. - Cloud = true tests run in the PR's Integration Tests check; that check is the evidence. Neither skill asks for a run link, an environment, or a statement about the run. - Decisions and API requests live in the code: the comment on the rule or mask, the test list entry, the exclusion line. The description is read for intent only and its completeness is not reviewed. - Repetition that matches a sibling is the expected result of copying the exemplar. The review skill gets a P10 pass that compares shape with the sibling and asks for an existing helper by name, never for a new one. - The resource-level launch stage comes only from launchStageOverrides in bundle/internal/annotation/preview.go; cli.json carries stages on fields only. The readiness check now describes that as a CLI decision instead of an upstream ask. Co-authored-by: Isaac <no-reply@databricks.com>
andrewnester
left a comment
andrewnester
left a comment
Contributor
There was a problem hiding this comment.
Nits:
bundle/direct/dresources/feature_spec.go:23: "what gets sent and stored in the backend" — the sorted copy is what is sent to the API and persisted in the bundle state; the backend stores whatever it wants. "sent to the API and persisted in state" would be more precise.
janniklasrose
approved these changes
Nov 11, 2026
janniklasrose
left a comment
janniklasrose
left a comment
Member
There was a problem hiding this comment.
Approving. The design matches precedent on every point: no DoUpdate for an API without one, KeyedSlices keyed on real fields, no plan overrides, approvals on both deploy and destroy, adapter and fake well under the size of the last resources, and the Integration Tests check is green.
One thing worth doing before merge: the Azure/GCP Repls entry on resources/feature_specs/test.toml, so basic matches there the way bind now does. Everything else in my earlier comments is follow-up material.
andrewnester
approved these changes
Nov 11, 2026
Resolve the conflict in bundle/schema/jsonschema.json by regenerating it, and regenerate the feature spec PyDABs models under the merged codegen. Co-authored-by: Isaac <no-reply@databricks.com>
deco-sdk-tagging Bot
added a commit
that referenced
this pull request
Nov 11, 2026
## Release v1.26.0 ### AI Runtime * `air run` reuses its existing workspace connection for config validation instead of opening a second one, which saves a TCP and TLS handshake per submission. ([#7210](#7210)) * Speed up `databricks air run` for git-pinned `code_source` snapshots by compressing the archive in parallel, which substantially shortens packaging on large repositories. ([#7402](#7402)) * Point the `air run -h config` agent-skill hint at the stable `databricks-ai-runtime` skill. ([#7454](#7454)) ### Bundles * Add `DATABRICKS_BUNDLE_INIT_WORKSPACE_HOST_PROFILE`, which names a profile whose host the `workspace_host` template helper returns instead of the selected profile's host. The profile must have the selected profile's workspace ID. ([#7410](#7410)) * Fixed an explicit `false` on optional boolean fields (`variables.*.sensitive`, `artifacts.*.dynamic_version`, `bundle.deployment.lock.force`, `workspace.azure_use_msi`, `replace_existing` on postgres endpoints, databases and roles, and the `experimental.*` flags) being dropped, which hid it from `bundle validate -o json` and kept a target from overriding `true` with it. ([#7412](#7412)) * Stop sending `compute_size` in the app update mask when the bundle does not declare it, so a deploy no longer resets a manually resized app to the default size. ([#7461](#7461)) * `lifecycle.depends_on` can now target the grants or permissions of a resource, as `schemas.foo.grants`. ([#7435](#7435)) * Add support for the `feature_specs` resource type, including UC grants on a feature spec. ([#7364](#7364)) ### Dependency Updates * Update the Go SDK to [v0.197.0](https://github.com/databricks/databricks-sdk-go/releases/tag/v0.197.0). ([#7457](#7457)) * Bump Go toolchain to 1.27.4. ([#7458](#7458))
Member
Author
|
@janniklasrose @andrewnester Thank you very much for approving and providing the thorough feedbacks! I opened a followup PR that addresses the proposed feedbacks and changes. |
idelafuente-oc
pushed a commit
to idelafuente-oc/cli
that referenced
this pull request
Nov 14, 2026
## Changes Addresses the review feedback on databricks#7364 that was still open at merge. Each item links the comments it answers. - **Approval prompts** ([Jan](databricks#7364 (comment)), [Jan's review](databricks#7364 (review))): the deploy prompt also names what a deleted FeatureSpec breaks, and both prompts say "FeatureSpecs", like the Feature View prompts. - **SQL stubs** ([Jan](databricks#7364 (comment)), [Andrew](databricks#7364 (comment))): the test server accepts `POST /api/2.0/sql/statements/` with the trailing slash the scripts send, and answers `CREATE TABLE`, `CREATE FUNCTION` and `INSERT INTO` with an empty `SUCCEEDED` result, as a fallback that a test's own matcher overrides. The `[[Server]]` stubs are gone from the feature spec, grants, bind, invariant and `features/lifecycle` tests. - **Shared fixture helpers** ([Andrew](databricks#7364 (comment))): the feature spec schema setup and cleanup move to `acceptance/bundle/script.prepare`, which the feature spec, grants, bind and invariant tests use. - **Shared statement runner** ([Andrew](databricks#7364 (comment))): `create_table.py` and `create_feature_spec_fixtures.py` share `run_statements` in `acceptance/bin/util.py`, so `create_table.py` now also waits for each statement and fails if one does not succeed. - **`resolveThreePartName`** ([Andrew](databricks#7364 (comment))): the vector search index, feature and feature spec blocks of `capture_uc_dependencies` share one helper. - **`?w=` in summary URLs** ([Jan](databricks#7364 (comment)), [Andrew](databricks#7364 (comment))): `resources/feature_specs/test.toml` strips the suffix, as `bind` does. - **Generated commands** ([Jan](databricks#7364 (comment))): the bind and remote-delete scripts use `feature-engineering create-feature-spec` and `delete-feature-spec` instead of `databricks api`. - **`nostamp`** ([Andrew](databricks#7364 (comment)), [Jan's review](databricks#7364 (review))): the grants test pipes plan JSON through `nostamp`. - **`TestAll` fixture** ([Jan](databricks#7364 (comment))): sets every feature spec field. - **Fake validation** ([Jan](databricks#7364 (comment))): the test server's feature spec create no longer checks for a three-part name. - **PyDABs test** ([Jan's review](databricks#7364 (review))): `python/feature_specs-support` runs locally only and has no `grants`, per `acceptance/bundle/python/README.md`. ## Why Review feedback on databricks#7364 that was left for after the merge. ## Tests - Acceptance tests: - `bundle/resources/feature_specs`, `bundle/resources/grants/feature_specs`, `bundle/deployment/bind/feature_spec`: re-recorded for the prompts, commands and URL - `bundle/invariant`, `bundle/resources/features/lifecycle`: pass without the SQL stub - `bundle/python/feature_specs-support`: local only, without grants - Unit tests: - `libs/testserver/testsql`: a fallback yields to a test's matcher - `libs/testserver`: default handlers accept fixture statements on both paths - `bundle/direct/dresources`: `TestAll/feature_specs` with every field set --------- Co-authored-by: Isaac <no-reply@databricks.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
Add
feature_specsas a bundle resource type, with agrantsfield for Unity Catalog privileges on the feature spec.IMMUTABLE. Any change therefore recreates the spec, and the adapter has noDoUpdate.grantsreuses the grants sub-resource with securable typefunction. When the spec is recreated, the bundle re-applies its declared grants to the new spec.rename_outputs,default_valuesandinput_bindingsof feature spec are compared by key. The string list fieldsexclude_columnsandextra_pip_requirements.requirementsare sorted in both the config and the read. This ensures reordering any of these order-free lists plans no change instead of a recreate.bundle deployandbundle destroyask for approval before recreating or deleting a feature spec.Why
Feature specs can only be created through the Python client or the API today. Supporting them in bundles is part of bringing the Feature Engineering API to Declarative Automation Bundles.
Tests
bundle/resources/feature_specs/basic: create, no-op plan, recreate, destroybundle/resources/feature_specs/order: which list reorders recreatebundle/resources/feature_specs/remote-delete: spec deleted outside the bundlebundle/resources/grants/feature_specs: grants re-applied after recreatebundle/deployment/bind/feature_spec: bind and unbind an existing specbundle/resource_deps/implicit_deps_feature_spec: deploys after its catalog and schemabundle/invariant/configs/feature_spec.yml.tmpl: no drift, repeatable delete and destroybundle/python/feature_specs-support: specs defined and changed in Pythonexperimental/open: listsfeature_specsbundle/direct/dresources/feature_spec_test.go: list sorting and matching