Skip to content

[arrow] Fix second-precision timestamp conversion for pre-epoch values - #9600

Open
thswlsqls wants to merge 1 commit into
apache:masterfrom
thswlsqls:fix/arrow-pre-epoch-second-timestamp-floor
Open

[arrow] Fix second-precision timestamp conversion for pre-epoch values#9600
thswlsqls wants to merge 1 commit into
apache:masterfrom
thswlsqls:fix/arrow-pre-epoch-second-timestamp-floor

Conversation

@thswlsqls

Copy link
Copy Markdown
Contributor

Purpose

fix #9599

  • ArrowUtils.nonCastedTimestampToEpoch() computed the epoch second for precision == 0 as timestamp.getMillisecond() / 1000. Java integer division truncates towards zero, so a pre-epoch value with sub-second millis moves forward across the epoch boundary: -500 (1969-12-31T23:59:59.500) becomes 0, i.e. 1970-01-01T00:00:00Z.
  • Use Math.floorDiv so truncation goes towards negative infinity. The 1-3, 4-6 and 7-9 precision branches are already negative-safe and are untouched.
  • This aligns the write direction with the read direction, fixed the same way in [arrow] Fix nanosecond timestamp conversion for pre-epoch values #9299 (Arrow2PaimonVectorConverter.convertEpochToTimestamp).

Tests

  • Added ArrowFormatWriterTest#testWritePreEpochSecondPrecisionTimestamp: an ArrowFormatWriter -> ArrowBatchReader round trip over TIMESTAMP(0) and TIMESTAMP_LTZ(0). It fails on master (expected: "1969-12-31T23:59:59" but was: "1970-01-01T00:00") and passes with the fix.
  • mvn -pl paimon-arrow clean install — 71 tests passed, 0 failures.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Arrow writer shifts pre-epoch second-precision timestamps forward

1 participant