Repository navigation
air: Name the YAML path in unknown config field errors - #7528
abdulkareem-db wants to merge 4 commits into
Conversation
Unknown or misspelled keys in an `air run` workload YAML surfaced the raw go-yaml error, which names internal Go types (e.g. "field rot_path not found in type aircmd.codeSourceConfig") and gives no path or suggestion. Walk the parsed YAML against configSchema() before decoding and report every unknown key with its line, dotted YAML path, a did-you-mean suggestion, the valid sibling fields, and the matching `-h config.<path>` pointer. When a key has no close match, note that it may come from newer documentation than the running CLI version supports. KnownFields(true) stays as a backstop. Fixes CNXT-2566 and CNXT-2554. Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
✅ Integration test reportCommit: 7879917
Top 6 slowest tests (at least 2 minutes):
|
…-field check Unknown keys inside an --override value were reported with the override string's line number as if they came from the config file. Validate each override value against its target schema field before splicing it in, and report it in the --override error voice. Only walk YAML sequences for list-typed fields (and mappings for non-list fields), so a shape mismatch such as a list document or `compute: [...]` surfaces the decoder's type error instead of "unknown field" plus the update-CLI hint. Merge keys with a list of sources are still walked. Hedge the newer-CLI hint, since some unknown keys (priority, _bases_) are intentionally unsupported, and extract visibleConfigChildNames for the three places that listed documented sibling fields. Co-authored-by: Isaac <no-reply@databricks.com>
pietern
left a comment
There was a problem hiding this comment.
@abdulkareem-db Please check out how we use libs/dyn for bundles. It implements the same thing and includes line/col information for every element in the object.
pardis-beikzadeh-db
left a comment
There was a problem hiding this comment.
Crash: collectUnknownConfigKeys runs before the decoder and doesn't track the aliases it has already visited. A mapping that merges its own anchor crashes air run with a Go stack overflow:
experiment_name: x
compute:
accelerator_type: GPU_1xA10
num_accelerators: 1
command: python t.py
code_source: &c
root_path: src
<<: *cruntime: goroutine stack exceeds 1000000000-byte limit
fatal error: stack overflow
On main the same file fails cleanly with yaml: anchor 'c' value contains itself. Nested merges also skip the decoder's alias limit. With 6 levels of 10 merges each, this PR reports 1,000,001 unknown fields in about 2s, where main rejects the file with excessive aliasing. Running the decoder first, or keeping a set of visited alias nodes, would fix both.
+1 to pietern's libs/dyn comment. One thing to know: dyn reads json tags, and the air config structs only have yaml tags.
Nits:
- runconfig_load.go:174 only checks whether the direct parent is hidden. For a key under
code_source.snapshot.git, the error says to rundatabricks air run -h config.code_source.snapshot.git, and that command fails withunknown config field "config.code_source.snapshot". - A single error names the field as
"code_source.rot_path"and then as"config.code_source". With this PR there are three wordings for an unknown field (-h, the file check, and the--overridepath). Could this reuseunknownConfigFieldError? - runconfig_test.go:379: no code prints
update the Databricks CLI, so this deny check can never fail. Should it check fornewer documentationinstead? - The description still says "
--overrideerrors are unchanged", but the last commit checks override values. Its "No close match" sample also doesn't match the hint text the code prints.
Changes
Unknown or misspelled keys in an
air runworkload YAML now produce an error that names the YAML path and line, suggests the closest valid field, lists the valid fields at that level, and points at the matching-h config.<path>help. When a key has no close match, the error also says it may come from newer documentation than this CLI version supports.Before:
After:
No close match:
Why
The go-yaml error exposes internal Go type names (
aircmd.runConfig,aircmd.codeSourceConfig) and gives no path or suggestion. A common way to hit it is running a current workload YAML on an older CLI. For example, the flatcode_source.root_pathform is rejected by CLIs before v1.20.0 and nothing in the error suggests upgrading.The check walks the parsed YAML against
configSchema(), the same schema that--overridepath validation and-h configuse. It reusesfindConfigChildandclosestConfigField, so hidden and legacy fields (code_source.type,code_source.snapshot) and inline fields are handled the same way. All unknown keys are reported, not just the first.KnownFields(true)stays as a backstop.--overrideerrors are unchanged.Tests
cmd/air/runconfig_test.go: a top-level typo, a nested typo, an unknown key with no close match (version hint), several unknown keys reported together, line numbers, and hidden legacy keys still loading.go test ./cmd/air/...passes.air run --dry-run: the three error cases above, and an unchanged valid workload config still validates.This pull request and its description were written by Isaac.