[core] Fix schema evolution for multiset elements - #9545
Conversation
JingsongLi
left a comment
There was a problem hiding this comment.
[P1] Support reading existing multiset data after committing the schema change
This change lets SchemaManager commit a multiset element-type evolution, but the read-side schema-evolution path still has no MultisetType support. SchemaEvolutionUtil.createCastExecutor handles RowType, ArrayType, and MapType; a multiset change falls through to CastExecutors.resolve, which returns null for multiset-to-multiset casts.
I reproduced this on the current PR head with the following sequence:
- Create a table containing
MULTISET<INT>. - Write a row under that schema.
- Alter the element type to
BIGINT. - Read the existing row.
The ALTER succeeds, but the read fails with:
NullPointerException: Cannot cast from type MULTISET<INT> to type MULTISET<BIGINT>
at SchemaEvolutionUtil.createCastExecutor(SchemaEvolutionUtil.java:262)
The same missing read path also affects nested row evolution inside a multiset, such as adding a field or widening an existing field, whenever files written with the old schema are present.
Please add a MultisetType branch to the read-side evolution logic. Since a multiset is represented as an InternalMap, it should recursively cast the key array (the elements) while preserving the integer multiplicity value array. An end-to-end regression test should write data before the schema change and read it afterward; the current tests only verify the resulting metadata.
Purpose
Tests