Skip to content

chore: bump avro to 209a373 to preserve adjust-to-utc on primitives - #954

Merged
wgtmac merged 1 commit into
apache:mainfrom
manuzhang:chore/bump-avro-4351
Sep 22, 2026
Merged

wgtmac merged 1 commit into
apache:mainfrom
manuzhang:chore/bump-avro-4351

Conversation

@manuzhang

Copy link
Copy Markdown
Member

What

Moves the avro-cpp pin in cmake_modules/IcebergThirdpartyToolchain.cmake from 997d50d to 209a373 (current apache/avro main, 74 commits ahead). The C++ changes in the range:

  • AVRO-4351 (209a373): avro-cpp now keeps custom attributes on primitive schema nodes when parsing and printing schema JSON.
  • 6444e8d: std::formatter for avro::Name/avro::Type templated on the format context, fixing C++23 builds.
  • cc134b3: JSON decoder rejects the lone low surrogate U+DFFF.

Adds tests that pin AVRO-4351:

  • AvroSchemaProjectionTest.ProjectTimestampTzFromParsedAdjustToUtc / RejectTimestampFromParsedAdjustToUtc — parse adjust-to-utc from schema JSON and project timestamptz, timestamptz_ns and timestamp against it.
  • AvroReaderParameterizedTest.TimestampTzTypes — write timestamptz and timestamptz_ns columns to a file and read them back, in both decoder modes.
  • AvroWriterTest.WriteTimestampTzTypes — write all four timestamp types, assert adjust-to-utc is present on each primitive node in the physical file schema, and read the data back, in both encoder modes.

Adds a comment on GetAdjustToUtc() recording why the pin matters.

Why

GetAdjustToUtc() reads adjust-to-utc off the primitive Avro node, and ValidateAvroSchemaEvolution gates timestamp vs timestamptz on it. The reader takes its file schema from the Avro file header, which avro-cpp parses from JSON. Before AVRO-4351, Compiler.cc only collected custom attributes for records, arrays and maps, and NodePrimitive::printJson never emitted them. So the attribute was lost twice: the writer dropped it from the file header, and the reader dropped it when parsing. Every timestamp read from a file then looked like it had no timezone, and projecting a timestamptz or timestamptz_ns column out of an Avro file failed with Cannot read Iceberg type: timestamptz from Avro type: ....

No test wrote and re-read a timestamptz column through a real file — the existing round trips use TimestampType only, and the timestamptz cases in avro_data_test.cc build the node in memory with ToAvroNodeVisitor — which is why this went unnoticed.

Behavior change

  • timestamptz and timestamptz_ns columns can now be read from Avro files.
  • Avro files written by iceberg-cpp now carry adjust-to-utc on timestamp primitives in the file header schema, as the Iceberg spec requires.

Testing

Built locally with gcc 15 at C++23 against the new pin and ran avro_test: 244/244 pass, including the 6 new tests (parameterized variants included). Rebuilt against the old pin 997d50d: the same 6 tests fail — reader/projection tests with Cannot read Iceberg type: timestamptz ..., writer test with node->customAttributes() being 0 — confirming they pin the fix. clang-format --dry-run -Werror and git diff --check are clean on the changed files.

🤖 Generated with Claude Code

Move the avro-cpp pin from 997d50d to 209a373 (current main, 74 commits).
The C++ changes in the range are a std::formatter fix for C++23 builds
(#3815), a JSON decoder fix for the lone low surrogate U+DFFF (#3841),
and AVRO-4351, which makes avro-cpp keep custom attributes on primitive
schema nodes when parsing and printing schema JSON.

AVRO-4351 is what matters here. `GetAdjustToUtc()` reads "adjust-to-utc"
off the primitive node, and the reader takes its file schema from the Avro
file header. With the old pin the attribute was dropped both when the
writer printed the header and when the reader parsed it, so every
timestamp read from a file looked like it had no timezone and projecting
a `timestamptz` or `timestamptz_ns` column out of an Avro file failed with
"Cannot read Iceberg type". No test wrote and re-read a timestamptz
column through a real file, so this went unnoticed.

Add schema-level projection tests that parse "adjust-to-utc" from JSON,
and file-level round trips that write timestamptz columns, assert the
attribute is present in the physical file schema, and read them back.
All of them fail on the old pin and pass on the new one.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Copilot AI lite review requested due to automatic review settings September 22, 2026 02:51

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@manuzhang
manuzhang requested a review from wgtmac September 22, 2026 02:54
@wgtmac

wgtmac commented Sep 22, 2026

Copy link
Copy Markdown
Member

Thanks for the quick update!

@wgtmac
wgtmac merged commit 64dc66f into apache:main Sep 22, 2026
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants