Skip to content

[format] Fail fast when skipping DELTA_LENGTH_BYTE_ARRAY data past end of page - #9571

Merged
JingsongLi merged 1 commit into
apache:masterfrom
LuciferYang:fix/skipbinary-eof-loop
Sep 4, 2026
Merged

[format] Fail fast when skipping DELTA_LENGTH_BYTE_ARRAY data past end of page#9571
JingsongLi merged 1 commit into
apache:masterfrom
LuciferYang:fix/skipbinary-eof-loop

Conversation

@LuciferYang

Copy link
Copy Markdown
Contributor

Purpose

close #9570

VectorizedDeltaLengthByteArrayReader.skipBinary walked each value's declared length like this:

int remaining = lengthsVector.getInt(currentRow + i);
while (remaining > 0) {
    remaining -= in.skip(remaining);
}

On a truncated or corrupt page the declared length exceeds the bytes left, and ByteBufferInputStream.skip returns -1 once the stream is dry, so remaining grows by one per iteration. The loop ends only when the int wraps negative, about 2^31 iterations, measured at 0.9s per value, and then the method returns with no error at all. The page stream is empty by then, so a later read on the same page reports Failed to read N bytes against a value that is not the corrupt one, and if the skip was the page's last operation nothing is reported at all. Reading the same page instead of skipping it fails cleanly and immediately, since readBinary and getBytes call in.slice(length) and convert the EOFException; the skip path was the only one that did not.

The fix uses in.skipFully(length), which skips or throws EOFException, matching those two methods and parquet-mr's own DeltaLengthByteArrayValuesReader.skip. That alone would regress on a negative declared length: SingleBufferInputStream.skip computes Math.min(remaining, n) and would move the position backwards, where the old loop did nothing while currentRow advanced, misaligning the reader silently. So a negative length is rejected explicitly. The exception carries its cause because the EOFException message says how far the skip got, which the wrapper text drops.

Tests

Both go in the existing DeltaLengthByteArrayEncodingTest, which already covers this reader including two skip cases. One writes the page normally and then drops three bytes off the data section, so the lengths still describe more data than the page holds. The other writes a length section that decodes to -1 through DeltaBinaryPackingValuesWriterForInteger, since no writer produces that.

Against the unfixed reader both fail with "Expected ParquetDecodingException to be thrown, but nothing was thrown", the truncation one after spinning for 0.9s. That is also the evidence for the wraparound described above.

mvn -pl paimon-format test on JDK 8: 598 tests, 0 failures. spotless:check and checkstyle:check are clean.

Not included, to keep this to one thing: VectorizedPlainValuesReader.skipBinary is a bare in.skip(len) that ignores the return value too. Its symptom is different, a silent under-skip rather than a spin, and PLAIN is the common encoding, so making it throw would turn files that read today, misaligned, into hard failures. That seems worth its own discussion.

…d of page

skipBinary walked each value's declared length with
"while (remaining > 0) { remaining -= in.skip(remaining); }". On a
truncated or corrupt page the declared length exceeds the bytes left, and
ByteBufferInputStream.skip returns -1 once the stream is dry, so
remaining grows by one per iteration until the int wraps negative: about
2^31 iterations, measured at 0.9s per value, and then the method returns
without an error. The page stream is empty by then, so a later read on
the same page reports "Failed to read N bytes" against a value that is
not the corrupt one, and if the skip was the page's last operation
nothing is reported at all. readBinary and getBytes turn the same input
into ParquetDecodingException directly, via in.slice.

Use in.skipFully, which skips or throws EOFException, matching those two
methods and parquet-mr's own DeltaLengthByteArrayValuesReader.skip. That
alone would regress on a negative declared length: SingleBufferInputStream
computes Math.min(remaining, n) and would move the position backwards,
where the old loop did nothing while currentRow advanced, misaligning the
reader silently. Reject a negative length explicitly. The exception
carries its cause because the EOFException message says how far the skip
got, which the wrapper text drops.

Tests go in the existing DeltaLengthByteArrayEncodingTest: one truncates
the data section of a written page, one writes a length section that
decodes to -1. Against the unfixed reader both fail with nothing thrown,
the first after spinning for 0.9s.

Assisted-by: GLM-5.3
@JingsongLi

Copy link
Copy Markdown
Contributor

+1

@JingsongLi
JingsongLi merged commit 1d5cc31 into apache:master Sep 4, 2026
12 of 13 checks passed
@LuciferYang

Copy link
Copy Markdown
Contributor Author

Thank you @JingsongLi

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Skipping past the end of a DELTA_LENGTH_BYTE_ARRAY page spins ~2^31 times and then reports nothing

2 participants