feat(databricks): accept Python model environment_key on serverless submit - #15968
Open
sd-db wants to merge 3 commits into
Open
feat(databricks): accept Python model environment_key on serverless submit#15968sd-db wants to merge 3 commits into
sd-db wants to merge 3 commits into
Conversation
…ubmit Fusion treated YAML environment_key and environment_dependencies as unused config (dbt1060). Recognize those keys and pass them through Jobs submit the same way dbt-databricks does. Fixes dbt-labs#15966
v1 treats [] as missing and still auto-builds the serverless environment spec. Match that, and keep the submit path a bit simpler.
![codescene-delta-analysis[bot]](https://avatars.githubusercontent.com/in/53921?s=60&v=4){kind=small}
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
Drop the boolean environments helper and the extract_packages wrapper. Reuse build_notebook_task in workflow submit without applying environment_key.
There was a problem hiding this comment.
Gates Failed
(2 files with Large Method, Complex Method)
Our agent can fix these. Install it.
Gates Passed
Reason for failure
| Enforce advisory code health rules | Violations | Code Health Impact | |
|---|---|---|---|
| manifest_nodes.rs | 1 advisory rule | 6.76 → 6.74 | Suppress |
| model_config.rs | 2 advisory rules | 2.60 → 2.59 | Suppress |
Quality Gate Profile: Clean Code Collective
Install CodeScene MCP: safeguard and uplift AI-generated code. Catch issues early with our IDE extension and CLI tool.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #15781
Upstream: databricks/dbt-databricks#1636
Problem
Fusion parse rejects Databricks Python model YAML
config.environment_keyandenvironment_dependenciesas unused (UnusedConfigKey, dbt1060). dbt-databricks already consumes those keys on serverless Jobs submit.Solution
environment_keyandenvironment_dependencieson Fusion model config (schema.yml anddbt_project.yml).environment_keyand auto-build job-levelenvironmentswhen dependencies are set (environment_version: "4"), matching dbt-databricks.python_job_config.environments, keep that value. Empty[]is treated as unset, same as v1.Testing
Ran a serverless Python model (
environment_key+environment_dependencies) live on Databricks with both dbt-databricks and Fusion. Parse no longer raisesUnusedConfigKey. Create and rerun both succeeded, the Python model table matched, andjobs/runs/get?include_resolved_values=truereturned the same resolved spec on both engines (environment_key,environment_version: "4", dependencies).Unit tests cover YAML key recognition (including unused-key behavior) and the Jobs submit payload (auto-build, user
environmentsoverride, empty[]).Checklist