[Bug] JSON format accepts unsupported types nested inside ARRAY/MAP/ROW at create time

[LuciferYang]

Search before asking

• I searched in the issues and found nothing similar.

Paimon version

master, 2788fe596 (2.1-SNAPSHOT).

Compute Engine

Flink and Spark, creating a managed table with 'file.format' = 'json'.

Minimal reproduce step

Create a JSON table with an unsupported type nested inside a container:

CREATE TABLE t (v ARRAY<VARIANT>) WITH ('file.format' = 'json');

CREATE TABLE succeeds. The failure comes later: on write, JsonFormatWriter reaches CastExecutors.resolveToString and throws Cast VARIANT to StringType is not supported., and on read JsonFileReader.convertPrimitiveStringToType gets a null cast executor and dereferences it, so the query fails with a NullPointerException carrying no message.

JsonFileFormat.validateDataType only ever looks at the outermost type root, because the four container cases fall through to a single break:

case ARRAY:
case VECTOR:
case MAP:
case ROW:
    // All types are supported in JSON
    break;
default:
    throw new UnsupportedOperationException("Unsupported data type for JSON format: " + dataType);

An unsupported type at the top level is rejected at create time with that message; the same type one level down is not.

What doesn't meet your expectations?

The other formats check nesting: ORC through OrcTypeUtil.convertToOrcType, Parquet through ParquetSchemaConverter.convertToParquetType and Avro through AvroSchemaConverter.convertToSchema all walk into element, key, value and field types and reject what they cannot represent. JSON was written like CSV, whose whitelist has no container types at all, but with the container types added to it, so the recursion was never there.

The comment saying all types are supported is not accurate either: the whitelist has always rejected MULTISET, VARIANT, BLOB, GEOMETRY and GEOGRAPHY at the top level.

Anything else?

This is about when the error arrives, not about data being lost: none of those five types has a cast rule to or from string, so they already fail at runtime today. Note also that validateDataFields is only reached for managed tables; a format table is created through CatalogUtils.validateCreateTable, which does not call it.

Are you willing to submit a PR?

• I'm willing to submit a PR!