datalake_fdw: read and write Parquet through Arrow - #1951
Open
MisterRaindrop wants to merge 1 commit into
Open
Conversation
MisterRaindrop
force-pushed
the
feature/datalake-parquet
branch
from
September 3, 2026 07:20
372580c to
bd3f0bb
Compare
MisterRaindrop
marked this pull request as ready for review
September 3, 2026 09:44
The format layer had an interface and no implementation. This is the Parquet one, and the conversions either side of it: PostgreSQL tuples into an Arrow batch, and an Arrow batch back into Datums. Arrow rather than libparquet alone, because libparquet is written in terms of Arrow's types, so linking one links the other. It is a build dependency now, found with pkg-config -- and whatever -std= its .pc file asks for is filtered out, because those flags land after CXXFLAGS and the Arrow project's own packages say -std=c++11. A fragment is a range of row groups rather than a whole file, so several segments can read one large file. Reading is single-threaded: a worker thread that fails has no way to report it through PostgreSQL. A reader and a writer hold a descriptor and memory from Arrow's allocator, which transaction abort does not reclaim, so both register with the resource owner -- the shape PAX uses in comm/pax_resource.cc. Types: bool, the four integers, both floats, text, varchar, char, bytea, date, timestamp, timestamptz. Anything else is refused by name. The 30-year epoch shift is range checked in both directions: PostgreSQL's range runs past what Arrow holds as microseconds from 1970, and a round trip through two unchecked halves would agree with itself. datalake_fdw_test is a second extension in the same library with the two functions this can be run with from SQL. Its regression case round-trips every supported type and checks that reading row groups separately gives back what reading the whole file does. Arrow 9.0.0 (EPEL 9) builds and passes; 17.0.0 (Rocky 10, gcc 14) and 17.0.0 and 21.0.0 (Rocky 8, gcc 8) compile without warnings. What is written here reads back correctly in pyarrow 21, which is not the implementation that wrote it.
MisterRaindrop
force-pushed
the
feature/datalake-parquet
branch
from
September 3, 2026 10:15
bd3f0bb to
777ba63
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
contrib/datalake_fdwlanded in #1842 with a format layer that had noimplementation. This adds the Parquet one, and the conversions either side of
it: PostgreSQL tuples into an Arrow batch, and an Arrow batch back into Datums.
Arrow rather than libparquet alone, because libparquet is written in terms of
Arrow's types, so linking one links the other.
Decisions worth a look:
can read one large file -- the granularity the parallel scan will need.
report it through PostgreSQL's error handling.
never calls a PostgreSQL allocator; the reader is C and reads the buffers
directly, so an allocation failure unwinds through C frames only.
transaction abort does not reclaim, so both register with the resource
owner -- the shape PAX uses in
comm/pax_resource.cc.range runs past what Arrow holds as microseconds from 1970, and a round trip
through two unchecked halves would agree with itself.
Types:
bool, the four integers, both floats,text,varchar,char,bytea,date,timestamp,timestamptz. Anything else is refused by name.Type of Change
Test Plan
datalake_fdw_testis a second extension in the same library with two functions-- write a query's result to a file, read a file back as rows. Until the access
method is finished there is no other way to run this layer in a real backend.
format_parquetcategory that round-tripsevery supported type, nulls and values on both sides of 1970 included, and
checks that reading row groups 0, 1 and 2 separately gives back what
reading the whole file does. Plus the refusals: unsupported column type,
wrong type on read, column count mismatch, a range past the end of the
file, a timestamp Arrow cannot hold, a bad row group size.
make installcheck-- 4/4, on a three-segment cluster with themodule preloaded.
make -C src/test installcheck-cbdb-parallel(not run)Beyond the suite:
wrote it (9.0.0). The Arrow schema is deliberately not stored in the file, so
what comes back is what any other reader sees rather than a note of our own.
and 17.0.0 and 21.0.0 on Rocky 8 with gcc 8, compile without warnings --
compile-only checks, not test runs.
Impact
Dependencies: new build dependency on the Arrow and Parquet C++ libraries.
The module is off by default and not in the RPM, so packaging is unchanged; the
CI job that builds it with PGXS installs them -- from EPEL on Rocky 9 and 10,
and from the Arrow project's own repository pinned to 17.0.0 on Rocky 8, where
EPEL's
libarrow-develcannot be installed (itsutf8proc-develismodular-filtered out of PowerTools) and the newest Arrow wants C++20.
Arrow's
.pcfile asks for-std=c++11, and pkg-config's cflags land afterCXXFLAGS, so it is filtered out. Otherwise Arrow's headers fail to compileagainst themselves, in a way that reads like the library needing a newer
compiler.
User-facing changes: one setting,
iceberg.batch_rows. Nothing else isreachable yet -- the access method still refuses anything that would touch data.
Checklist
datalake_parquet_write(path, query)names a path on the server's file systemand runs a query through SPI, so it is as privileged as
pg_read_server_filesand granted the same way:
REVOKE EXECUTE ... FROM PUBLIC, superuser only.Additional Context
Left for later, deliberately rather than by oversight:
arrow::io::RandomAccessFileovercommon/file_system_wrapper.h, and the twoparquet files are the only ones that change when it does.
open_readerrefuses a filter set rather thanignoring it: ignoring it would still give the right rows, which is exactly why
a caller that believed pruning had happened could never find out.
change.
datalake_fdw_test's control file still lands in the share directory. PGXS'sNO_INSTALLis per-module, and these functions have to live in the librarywhose internals they test.