chore: bump avro to 209a373 to preserve adjust-to-utc on primitives - #954
Merged
Merged
Conversation
Move the avro-cpp pin from 997d50d to 209a373 (current main, 74 commits). The C++ changes in the range are a std::formatter fix for C++23 builds (#3815), a JSON decoder fix for the lone low surrogate U+DFFF (#3841), and AVRO-4351, which makes avro-cpp keep custom attributes on primitive schema nodes when parsing and printing schema JSON. AVRO-4351 is what matters here. `GetAdjustToUtc()` reads "adjust-to-utc" off the primitive node, and the reader takes its file schema from the Avro file header. With the old pin the attribute was dropped both when the writer printed the header and when the reader parsed it, so every timestamp read from a file looked like it had no timezone and projecting a `timestamptz` or `timestamptz_ns` column out of an Avro file failed with "Cannot read Iceberg type". No test wrote and re-read a timestamptz column through a real file, so this went unnoticed. Add schema-level projection tests that parse "adjust-to-utc" from JSON, and file-level round trips that write timestamptz columns, assert the attribute is present in the physical file schema, and read them back. All of them fail on the old pin and pass on the new one. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
wgtmac
approved these changes
Sep 22, 2026
Member
|
Thanks for the quick update! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Moves the avro-cpp pin in
cmake_modules/IcebergThirdpartyToolchain.cmakefrom997d50dto209a373(currentapache/avromain, 74 commits ahead). The C++ changes in the range:209a373): avro-cpp now keeps custom attributes on primitive schema nodes when parsing and printing schema JSON.6444e8d:std::formatterforavro::Name/avro::Typetemplated on the format context, fixing C++23 builds.cc134b3: JSON decoder rejects the lone low surrogate U+DFFF.Adds tests that pin AVRO-4351:
AvroSchemaProjectionTest.ProjectTimestampTzFromParsedAdjustToUtc/RejectTimestampFromParsedAdjustToUtc— parseadjust-to-utcfrom schema JSON and projecttimestamptz,timestamptz_nsandtimestampagainst it.AvroReaderParameterizedTest.TimestampTzTypes— writetimestamptzandtimestamptz_nscolumns to a file and read them back, in both decoder modes.AvroWriterTest.WriteTimestampTzTypes— write all four timestamp types, assertadjust-to-utcis present on each primitive node in the physical file schema, and read the data back, in both encoder modes.Adds a comment on
GetAdjustToUtc()recording why the pin matters.Why
GetAdjustToUtc()readsadjust-to-utcoff the primitive Avro node, andValidateAvroSchemaEvolutiongatestimestampvstimestamptzon it. The reader takes its file schema from the Avro file header, which avro-cpp parses from JSON. Before AVRO-4351,Compiler.cconly collected custom attributes for records, arrays and maps, andNodePrimitive::printJsonnever emitted them. So the attribute was lost twice: the writer dropped it from the file header, and the reader dropped it when parsing. Every timestamp read from a file then looked like it had no timezone, and projecting atimestamptzortimestamptz_nscolumn out of an Avro file failed withCannot read Iceberg type: timestamptz from Avro type: ....No test wrote and re-read a
timestamptzcolumn through a real file — the existing round trips useTimestampTypeonly, and thetimestamptzcases inavro_data_test.ccbuild the node in memory withToAvroNodeVisitor— which is why this went unnoticed.Behavior change
timestamptzandtimestamptz_nscolumns can now be read from Avro files.adjust-to-utcon timestamp primitives in the file header schema, as the Iceberg spec requires.Testing
Built locally with gcc 15 at C++23 against the new pin and ran
avro_test: 244/244 pass, including the 6 new tests (parameterized variants included). Rebuilt against the old pin997d50d: the same 6 tests fail — reader/projection tests withCannot read Iceberg type: timestamptz ..., writer test withnode->customAttributes()being0— confirming they pin the fix.clang-format --dry-run -Werrorandgit diff --checkare clean on the changed files.🤖 Generated with Claude Code