Repository navigation
[draft] Accept YAML arrays for for_each_task inputs - #7304
lennartkats-db wants to merge 8 commits into
Conversation
✅ Integration test reportCommit: 2e23b88
Top 6 slowest tests (at least 2 minutes):
|
andrewnester
left a comment
There was a problem hiding this comment.
A few cross-cutting asks:
bundle/schema/jsonschema.jsonstill typesfor_each_task.inputsas$defs/string, so the YAML array the CLI now accepts is flagged as invalid by schema-aware editors. The shadoweddefinition/serialized_dashboardfields get$defs/interfacefrom their Go type; here the Go type can't change, so the schema needs an explicit string-or-array. Can a per-type hook inbundle/internal/schema/main.go(likeremoveJobsFields, keyed onreflect.TypeFor[jobs.ForEachTask]()) add that, with the schema regenerated?- This is user-visible config behaviour with only unit coverage.
acceptance/bundle/resources/cluster_policies/definition_yamlis the template for the same feature on cluster policies:validate -o json,deploy,print_requests.pyon the create call,Cloud = true. The description says no live workspace run was done, and the array-of-objects case in particular relies on the Jobs API accepting that JSON. Can you add an equivalent test underacceptance/bundle/resources/jobs/withCloud = true, including the target-override and variable-reference cases? - No
.nextchanges/bundles/fragment; this is user-visible, so it needs one per.nextchanges/README.md.
| return dyn.InvalidValue, fmt.Errorf("%s at %s: %w", p, input.Location(), err) | ||
| } | ||
| // Leave unresolved references for normal validation instead of interpolating into JSON text. | ||
| if dynvar.ContainsVariableReference(string(data)) { |
There was a problem hiding this comment.
With inputs: ["${var.a}_x", "y"] and a plain string variable a, this branch hands the sequence back, convert.Normalize then warns expected string, found sequence and required field "inputs" is not set, and bundle validate exits 0 with inputs gone (checked against the fake server). The hand-written JSON string form with the same ${var.a} inside interpolates fine via ResolveVariableReferencesOnlyResources, so the array form is the only one that can't carry a reference, and the warning never mentions the reference. Can we marshal regardless and let the usual string interpolation run on the JSON text, as the string form already does, or else fail here with an error that names the reference instead of falling through to a type warning?
There was a problem hiding this comment.
I changed this to fail during loading when a YAML array contains an unescaped ${...} reference. The error names the reference and YAML location, so bundle validate exits nonzero. The array_reference acceptance fixture asserts that behavior. I kept string inputs for interpolation because substituting a quoted value after JSON encoding could change the payload.
-- Isaac
| })) | ||
| } | ||
|
|
||
| func rewriteForEachTaskInputs(v dyn.Value) (dyn.Value, error) { |
There was a problem hiding this comment.
ConfigureClusterPolicyDefinition and ConfigureDashboardSerializedDashboard in bundle/config/mutator/resourcemutator/ already do this job (inline YAML to a JSON string for an SDK string field) as resource mutators after variable resolution, with an error diagnostic for an unsupported kind. This one runs in the loader before normalization. I assume that is because jobs.Task.ForEachTask.Inputs can't be shadowed with any the way resources.ClusterPolicy.Definition is; can the doc comment on this function say that, and spell out how its behaviour differs from those mutators (timing relative to variables, no error on other kinds)?
There was a problem hiding this comment.
I expanded the loader comment to explain that a resource-level any shadow cannot cover this nested SDK field, so literal arrays are encoded before typed normalization. The cluster policy and dashboard mutators run after variable resolution. Other kinds still use normal validation.
-- Isaac
| }) | ||
| } | ||
|
|
||
| func isForEachTaskInputsPath(p dyn.Path) bool { |
There was a problem hiding this comment.
bundle/config/validate/single_node_cluster.go:117-119 matches this same task path with dyn.NewPattern(..., dyn.Key("tasks"), dyn.AnyIndex(), dyn.Key("for_each_task"), ...) and dyn.MapByPattern. Can this use patterns for the three prefixes (root, targets.*, environments.*) instead of the length and Key() != "" checks, so the path shape lives in the idiom the rest of the repo uses?
There was a problem hiding this comment.
I replaced the length and key checks with dyn.Pattern matches for root, targets.*, and environments.* paths. The root and target cases exercise the matched task path.
-- Isaac
| return input, nil | ||
| } | ||
|
|
||
| data, err := json.Marshal(input.AsAny()) |
There was a problem hiding this comment.
AsAny() turns a YAML timestamp into time.Time, so inputs: [2024-01-01, 2024-01-02] is sent as ["2024-01-01T00:00:00Z","2024-01-02T00:00:00Z"], while the same scalar outside an array keeps its literal through normalizeString (MustTime().String()). Is the reformat intended, or should time values be stringified the way normalize does before marshalling?
There was a problem hiding this comment.
I now convert dyn.KindTime nodes to their original MustTime().String() values before marshaling. The unit test covers date-only elements and a nested timestamp with fractional seconds and a timezone offset.
-- Isaac
| {name: "YAML object array", inputs: "[{name: input_1, enabled: true}]", want: `[{"enabled":true,"name":"input_1"}]`}, | ||
| {name: "empty YAML array", inputs: "[]", want: `[]`}, | ||
| {name: "JSON string", inputs: `'[ "input_1", "input_2" ]'`, want: `[ "input_1", "input_2" ]`}, | ||
| {name: "variable reference string", inputs: `${var.items}`, want: `${var.items}`}, |
There was a problem hiding this comment.
This case is a string, so it never reaches the sequence branch in rewriteForEachTaskInputs; the array-with-reference path (["${var.a}_x", "y"]) is untested, and today it would fail this test: inputs ends up empty with only a warning. Can you add that case so the expected behaviour for references inside the array is pinned down?
There was a problem hiding this comment.
I added tests for array references in a root element, a nested object, and a target override. The acceptance fixture confirms that bundle validate exits nonzero and names the reference, path, and source location. I also checked that an embedded reference in the existing JSON string form still reaches the Jobs create request with its interpolated value.
-- Isaac
|
Commit: Regression test reportSelected 3 changed tests (limit=20, 0 not selected) 6 tests: regression=4 unreleased=0 coverage=2 failing=0
TestAccept/bundle/resources/jobs/for_each_inputs/array_reference/DMS=: branch ✅ | main (8e70c8b) ❌ | latest (v1.26.0) ❌main (8e70c8b): latest (v1.26.0): same output as main (8e70c8b) TestAccept/bundle/resources/jobs/for_each_inputs/array_reference/DMS=true: branch ✅ | main (8e70c8b) ❌ | latest (v1.26.0) ❌main (8e70c8b): latest (v1.26.0): same output as main (8e70c8b) TestAccept/bundle/resources/jobs/for_each_inputs/literals/DMS=: branch ✅ | main (8e70c8b) ❌ | latest (v1.26.0) ❌main (8e70c8b): latest (v1.26.0): same output as main (8e70c8b) TestAccept/bundle/resources/jobs/for_each_inputs/literals/DMS=true: branch ✅ | main (8e70c8b) ❌ | latest (v1.26.0) ❌main (8e70c8b): latest (v1.26.0): same output as main (8e70c8b) |
Changes
for_each_task.inputs, including target overrides, and send their JSON encoding in the Jobs API string field.${...}references inside YAML arrays with an error naming the reference, path, and YAML location; use the JSON string form for interpolated inputs.Scope
This PR supports arrays authored directly at
for_each_task.inputs. It does not serialize array-valued complex variables such asinputs: ${var.items}, or arrays inside variable-defined task or job containers. Issue #6901 tracks that separate variable-resolution behavior. The variable-container fixture here checks that existing JSON string inputs still work through those containers.Why
Issue #2370 reports that bundle authors must hand-escape JSON arrays to configure for-each task inputs. The SDK models this field as a string, so typed normalization cannot accept a native YAML sequence. Encoding directly authored arrays during loading supports the requested syntax while retaining existing string behavior.
Fixes #2370
Tests
go test ./bundle/config ./bundle/schema ./bundle/internal/schema -count=1go test ./acceptance -run '^TestAccept/bundle/resources/jobs/for_each_inputs' -test.v -timeout=20magainst the local fake server, with both DMS modes. The fixture records the Jobs API create payloads.uv run --frozen -m pytestcould not install its dependencies locally because the PyPI connection reset while fetchingpygments.