Skip to content

[core] Fix schema evolution for multiset elements - #9545

Open
ArnavBalyan wants to merge 1 commit into
apache:masterfrom
ArnavBalyan:arnavb/fix-ms
Open

[core] Fix schema evolution for multiset elements#9545
ArnavBalyan wants to merge 1 commit into
apache:masterfrom
ArnavBalyan:arnavb/fix-ms

Conversation

@ArnavBalyan

Copy link
Copy Markdown
Member

Purpose

  • Schema evolution fails when updating nested fields or multiset elements.
  • Paimon generates the nested schema, but the schema manager does not traverse multiset elements and reports that the element column does not exist.
  • Add multiset traversal/reconstruction to apply nested field evolution.

Tests

  • UT

@JingsongLi JingsongLi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Support reading existing multiset data after committing the schema change

This change lets SchemaManager commit a multiset element-type evolution, but the read-side schema-evolution path still has no MultisetType support. SchemaEvolutionUtil.createCastExecutor handles RowType, ArrayType, and MapType; a multiset change falls through to CastExecutors.resolve, which returns null for multiset-to-multiset casts.

I reproduced this on the current PR head with the following sequence:

  1. Create a table containing MULTISET<INT>.
  2. Write a row under that schema.
  3. Alter the element type to BIGINT.
  4. Read the existing row.

The ALTER succeeds, but the read fails with:

NullPointerException: Cannot cast from type MULTISET<INT> to type MULTISET<BIGINT>
    at SchemaEvolutionUtil.createCastExecutor(SchemaEvolutionUtil.java:262)

The same missing read path also affects nested row evolution inside a multiset, such as adding a field or widening an existing field, whenever files written with the old schema are present.

Please add a MultisetType branch to the read-side evolution logic. Since a multiset is represented as an InternalMap, it should recursively cast the key array (the elements) while preserving the integer multiplicity value array. An end-to-end regression test should write data before the schema change and read it afterward; the current tests only verify the resulting metadata.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants