[format] Report unresolvable JSON string casts instead of a bare NPE - #9588
Merged
Conversation
convertPrimitiveStringToType's default branch dereferenced the result of CastExecutors.resolve, whose javadoc says it returns null when no rule can be resolved. For a column whose type has no cast from STRING, MULTISET, VARIANT or BLOB, the read failed with a NullPointerException carrying no message. Throw instead, with the type named. The exception and its wording match what the same format already says at create time, UnsupportedOperationException "Unsupported data type for JSON format: X" from JsonFileFormat.validateDataType, so both stages report the same thing. The other STRING-to-X resolve sites already do something sensible: CsvParser falls back to the raw string, and DefaultValueUtils and the StringToArray/Map/Row rules throw. Reaching it needs a format table. SchemaValidation is the only entry point for validateDataFields, so a managed table rejects these types at create time, while a format table is created through CatalogUtils.validateCreateTable, which does not call it. One limit worth stating: with json.ignore-parse-errors = true, handleParseError still swallows this like any other conversion failure, so the row is emitted with that column null. The improvement is to the default configuration. Making that path fail loudly means resolving the casts once when the reader is built, which is a change of its own. Assisted-by: GLM-5.3
Contributor
|
+1 |
Contributor
Author
|
Thank you @JingsongLi |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
close #9587
JsonFileReader.convertPrimitiveStringToType's default branch dereferenced the result ofCastExecutors.resolve, whose javadoc says it returns null when no rule can be resolved. For a column whose type has no cast from STRING,MULTISET,VARIANTorBLOB, the read failed with aNullPointerExceptioncarrying no message. This throws instead, naming the type.The exception and its wording are the ones the same format already uses at create time,
UnsupportedOperationException("Unsupported data type for JSON format: X")fromJsonFileFormat.validateDataType, so both stages report the same thing. The other STRING-to-X resolve sites already do something sensible:CsvParserfalls back to the raw string, andDefaultValueUtilsand theStringToArray/Map/Rowrules throw.Reaching it needs a format table.
SchemaValidationis the only entry point forvalidateDataFields, so a managed table rejects these types at create time, while a format table is created throughCatalogUtils.validateCreateTable, which does not call it.One limit worth stating plainly: with
json.ignore-parse-errors = true,handleParseErrorswallows this like any other conversion failure, so the row is emitted with that column null. This improves the default configuration. Making that path fail loudly means resolving the casts once when the reader is built, which is a change of its own.Tests
JsonFileFormatTest.testUnresolvableCastFailsWithClearMessagereads a one-line JSON file with aMULTISET<STRING>read type and asserts both the root cause type and its message. Asserting the type matters here, since the message alone would also pass if the thrown type changed, and the type is the point of the change.Against the unfixed reader the root cause is a
NullPointerExceptionwith no message, so the assertion fails on the type.mvn -pl paimon-format teston JDK 8: 597 tests, 0 failures.spotless:checkandcheckstyle:checkare clean.Adjacent and not included: the same unguarded
CastExecutors.resolvedereference exists inArrayToStringCastRule,MapToStringCastRule,RowToStringCastRuleandInternalRowPartitionComputer. I checked that they are unguarded, not that they are reachable, so they seem better handled separately.