Skip to content

datalake_fdw: read and write Parquet through Arrow - #1951

Open
MisterRaindrop wants to merge 1 commit into
apache:mainfrom
MisterRaindrop:feature/datalake-parquet
Open

datalake_fdw: read and write Parquet through Arrow#1951
MisterRaindrop wants to merge 1 commit into
apache:mainfrom
MisterRaindrop:feature/datalake-parquet

Conversation

@MisterRaindrop

@MisterRaindrop MisterRaindrop commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

What does this PR do?

contrib/datalake_fdw landed in #1842 with a format layer that had no
implementation. This adds the Parquet one, and the conversions either side of
it: PostgreSQL tuples into an Arrow batch, and an Arrow batch back into Datums.
Arrow rather than libparquet alone, because libparquet is written in terms of
Arrow's types, so linking one links the other.

Decisions worth a look:

  • A fragment is a range of row groups, not a whole file, so several segments
    can read one large file -- the granularity the parallel scan will need.
  • Reading is single-threaded: a worker thread that fails has no way to
    report it through PostgreSQL's error handling.
  • The Arrow C data interface is the C/C++ boundary. The writer is C++ and
    never calls a PostgreSQL allocator; the reader is C and reads the buffers
    directly, so an allocation failure unwinds through C frames only.
  • A reader and a writer hold a descriptor and Arrow-pool memory, which
    transaction abort does not reclaim, so both register with the resource
    owner
    -- the shape PAX uses in comm/pax_resource.cc.
  • The 30-year epoch shift is range checked in both directions: PostgreSQL's
    range runs past what Arrow holds as microseconds from 1970, and a round trip
    through two unchecked halves would agree with itself.

Types: bool, the four integers, both floats, text, varchar, char,
bytea, date, timestamp, timestamptz. Anything else is refused by name.

Type of Change

  • Bug fix (non-breaking change)
  • New feature (non-breaking change)
  • Breaking change (fix or feature with breaking changes)
  • Documentation update

Test Plan

datalake_fdw_test is a second extension in the same library with two functions
-- write a query's result to a file, read a file back as rows. Until the access
method is finished there is no other way to run this layer in a real backend.

  • Integration tests added -- a format_parquet category that round-trips
    every supported type, nulls and values on both sides of 1970 included, and
    checks that reading row groups 0, 1 and 2 separately gives back what
    reading the whole file does. Plus the refusals: unsupported column type,
    wrong type on read, column count mismatch, a range past the end of the
    file, a timestamp Arrow cannot hold, a bad row group size.
  • Passed make installcheck -- 4/4, on a three-segment cluster with the
    module preloaded.
  • Unit tests added/updated
  • Passed make -C src/test installcheck-cbdb-parallel (not run)

Beyond the suite:

  • pyarrow 21 reads what this writes, which is not the implementation that
    wrote it (9.0.0). The Arrow schema is deliberately not stored in the file, so
    what comes back is what any other reader sees rather than a note of our own.
  • Arrow 9.0.0 (EPEL 9) builds and passes. 17.0.0 on Rocky 10 with gcc 14,
    and 17.0.0 and 21.0.0 on Rocky 8 with gcc 8, compile without warnings --
    compile-only checks, not test runs.

Impact

Dependencies: new build dependency on the Arrow and Parquet C++ libraries.
The module is off by default and not in the RPM, so packaging is unchanged; the
CI job that builds it with PGXS installs them -- from EPEL on Rocky 9 and 10,
and from the Arrow project's own repository pinned to 17.0.0 on Rocky 8, where
EPEL's libarrow-devel cannot be installed (its utf8proc-devel is
modular-filtered out of PowerTools) and the newest Arrow wants C++20.

Arrow's .pc file asks for -std=c++11, and pkg-config's cflags land after
CXXFLAGS, so it is filtered out. Otherwise Arrow's headers fail to compile
against themselves, in a way that reads like the library needing a newer
compiler.

User-facing changes: one setting, iceberg.batch_rows. Nothing else is
reachable yet -- the access method still refuses anything that would touch data.

Checklist

datalake_parquet_write(path, query) names a path on the server's file system
and runs a query through SPI, so it is as privileged as pg_read_server_files
and granted the same way: REVOKE EXECUTE ... FROM PUBLIC, superuser only.

Additional Context

Left for later, deliberately rather than by oversight:

  • Local files only. Object storage arrives as an
    arrow::io::RandomAccessFile over common/file_system_wrapper.h, and the two
    parquet files are the only ones that change when it does.
  • No row group pruning. open_reader refuses a filter set rather than
    ignoring it: ignoring it would still give the right rows, which is exactly why
    a caller that believed pruning had happened could never find out.
  • No NUMERIC. DECIMAL has four storage forms in Parquet and deserves its own
    change.
  • No merge-on-read row ordinal yet; it arrives with positional deletes.
  • datalake_fdw_test's control file still lands in the share directory. PGXS's
    NO_INSTALL is per-module, and these functions have to live in the library
    whose internals they test.

@MisterRaindrop
MisterRaindrop force-pushed the feature/datalake-parquet branch from 372580c to bd3f0bb Compare September 3, 2026 07:20
@MisterRaindrop
MisterRaindrop marked this pull request as ready for review September 3, 2026 09:44
The format layer had an interface and no implementation.  This is the
Parquet one, and the conversions either side of it: PostgreSQL tuples
into an Arrow batch, and an Arrow batch back into Datums.

Arrow rather than libparquet alone, because libparquet is written in
terms of Arrow's types, so linking one links the other.  It is a build
dependency now, found with pkg-config -- and whatever -std= its .pc file
asks for is filtered out, because those flags land after CXXFLAGS and
the Arrow project's own packages say -std=c++11.

A fragment is a range of row groups rather than a whole file, so several
segments can read one large file.  Reading is single-threaded: a worker
thread that fails has no way to report it through PostgreSQL.  A reader
and a writer hold a descriptor and memory from Arrow's allocator, which
transaction abort does not reclaim, so both register with the resource
owner -- the shape PAX uses in comm/pax_resource.cc.

Types: bool, the four integers, both floats, text, varchar, char, bytea,
date, timestamp, timestamptz.  Anything else is refused by name.  The
30-year epoch shift is range checked in both directions: PostgreSQL's
range runs past what Arrow holds as microseconds from 1970, and a round
trip through two unchecked halves would agree with itself.

datalake_fdw_test is a second extension in the same library with the two
functions this can be run with from SQL.  Its regression case round-trips
every supported type and checks that reading row groups separately gives
back what reading the whole file does.

Arrow 9.0.0 (EPEL 9) builds and passes; 17.0.0 (Rocky 10, gcc 14) and
17.0.0 and 21.0.0 (Rocky 8, gcc 8) compile without warnings.  What is
written here reads back correctly in pyarrow 21, which is not the
implementation that wrote it.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant