[format] Close the ParquetFileReader when post-construction setup fails - #9576
Merged
JingsongLi merged 1 commit intoSep 4, 2026
Merged
Conversation
createReader already closes what it opened if reading the footer or constructing the ParquetFileReader throws. The stretch after that was not covered: building the shredding read plan, resolving the requested schema, checking the batch size, allocating the writable vectors and constructing VectorizedParquetRecordReader can all throw, and at that point the reader is only a local variable, so the caller cannot close it and the open stream is lost. Wrap that stretch too, using the same shape as the block above it. Measured as a diff that ignores whitespace it is 10 added lines; the rest of the churn is the indentation of the wrapped block. What makes it worth fixing is that DataFileRecordReader treats an IOException or RuntimeException from createReader as a corrupt file when scan.ignore-corrupt-files is on: it logs a WARN, returns null, and the scan continues, so each unreadable file costs one descriptor and nothing says so. Reaching it does not need a damaged file: reading a column whose file type does not match the read type throws "Schema evolution not supported." from VectorizedParquetRecordReader's constructor, and a case-insensitive read of two columns differing only in case throws from the requested-schema build. The test drives it deterministically by overriding computeBatchSize to return zero, which fails the positive-batch-size check inside that stretch. It counts closes rather than recording a boolean, so the second case pins the successful read at exactly one close, which is what a careless unconditional close would break. Assisted-by: GLM-5.3
Contributor
|
+1 |
Contributor
Author
|
Thank you @JingsongLi |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
close #9575
ParquetReaderFactory.createReadercloses what it opened if reading the footer or constructing theParquetFileReaderthrows. The stretch after that was not covered: building the shredding read plan, resolving the requested schema, checking the batch size, allocating the writable vectors and constructingVectorizedParquetRecordReadercan all throw, and at that point the reader is only a local variable, so the caller cannot close it and the open stream is lost. This wraps that stretch too, with the samecatch (Throwable)shape as the block directly above it, closingreaderinstead ofinputStream.Two notes on reading the diff. Ignoring whitespace it is 10 added lines, one comment plus
try {plus the eight-line catch; the rest of the churn is the indentation of the wrapped block, sogit diff -wis the useful view. And the two catches are mutually exclusive, so nothing is closed twice.What makes it worth fixing is that
DataFileRecordReadertreats anIOExceptionorRuntimeExceptionfromcreateReaderas a corrupt file whenscan.ignore-corrupt-filesis on: it logs a WARN, returns null and the scan continues, so each unreadable file costs one descriptor and nothing says so. Reaching it does not need a damaged file. A column whose stored type does not match the read type throwsSchema evolution not supported.fromVectorizedParquetRecordReader's constructor, and a case-insensitive read of two columns differing only in case throwsFound duplicate field(s)from the requested-schema build.Tests
ParquetReaderFactoryLeakTestoverridescomputeBatchSizeto return zero, which fails the positive-batch-size check inside the newly protected stretch. That is a deterministic trigger rather than a schema quirk, and it uses the method's documented extension point: the javadoc says subclasses may override it for per-file batch sizing, and thecheckArgumentexists to reject a non-positive result from exactly that.The
FileIOcountsclose()calls per stream rather than recording a boolean, so the second test pins a successful read at exactly one close. That one passes against the unfixed code as well, which is the point of it: it guards against a careless unconditional close. The leak test fails against the unfixed code on the close count.mvn -pl paimon-format clean teston JDK 8: 598 tests, 0 failures.spotless:checkandcheckstyle:checkare clean.No overlap with #9550, which touches
clipParquetTypeandParquetSchemaConverterrather thancreateReader.