[format] Close the text stream when line-reader construction fails - #9579
Merged
Conversation
AbstractTextFileReader's constructor opens the file, wraps it for decompression and then builds the line reader. Nothing else holds the stream until that last assignment, so if either step throws, the constructor never returns and the stream is gone. The reachable case is an unreadable compressed file: for a .gz or .zst path the reader wraps the stream in a Hadoop codec, StandardLineReader reads in its own constructor, and a bad header fails there. This is a plain read with the default delimiter and no offset, and it loses the descriptor plus the decompressor the codec borrowed from CodecPool. Close on the failure path. Closing the decompression wrapper closes the stream underneath it, and when the file is not compressed the wrapper is the stream itself, so this is one close in both cases rather than one per layer. TextLineReader.create also throws for a custom line delimiter with an offset, which is what the tests cover second. That one is defensive: SplitEnumerator.preferToSplitFile only splits CSV and JSON files that use the default delimiter and are not compressed, so no FileMeta with an offset ever reaches it. The tests count closes per stream rather than recording a boolean, so the third case pins a successful read at exactly one close. The two failure cases fail against the unfixed constructor. Assisted-by: GLM-5.3
Contributor
|
+1 |
Contributor
Author
|
Thank you @JingsongLi |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
close #9578
AbstractTextFileReader's constructor opens the file, wraps it for decompression and then builds the line reader. Nothing else holds the stream until that last assignment, so if either step throws, the constructor never returns and the stream is lost.The reachable case is an unreadable compressed file. The wrapper is chosen from the path suffix, so a compressed path such as
.gzor.zstgoes through a Hadoop codec,StandardLineReaderreads in its own constructor, and a bad header fails there. That is a plain read with the default delimiter and no offset, and it loses the descriptor plus the decompressor the codec borrowed fromCodecPool. It matters most underscan.ignore-corrupt-files, where the caller swallows the failure and continues to the next file.Closing the decompression wrapper closes the stream underneath it, and when the file is not compressed the wrapper is that same stream, so one
closeQuietlycovers both cases rather than one close per layer.TextLineReader.createalso throws for a custom line delimiter combined with an offset, which the tests cover second. That path is defensive rather than reachable:SplitEnumerator.preferToSplitFileonly splits CSV and JSON files that use the default delimiter and are not compressed, so noFileMetawith an offset is ever produced for one.Tests
TextReaderCtorLeakTestwrapsLocalFileIOso each stream counts its ownclose()calls, and asserts exactly one close rather than just "was closed": a corrupt.gz, the custom-delimiter-with-offset case, and a successful read. The two failure cases fail against the unfixed constructor. The success case passes there too, which is the point of it: it pins the single close, which a per-layer close would turn into two.mvn -pl paimon-format teston JDK 8: 599 tests, 0 failures.spotless:checkandcheckstyle:checkare clean.One related gap this does not cover:
CsvFileReaderbuilds itsCsvParseraftersuper(...)returns, andcreateProjectionMappingthrowsIllegalArgumentExceptionwhen a projected field is missing. The line reader exists by then, so that path leaks too, but fixing it belongs in the subclass rather than in this constructor.