Repository navigation
Add optional Arrow Java integration pinned to chdb-core v26.11.0-rc.3 - #37
Closed
ShawnChen-Sirius wants to merge 2 commits into
Closed
ShawnChen-Sirius wants to merge 2 commits into
ShawnChen-Sirius wants to merge 2 commits into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add the optional
com.clickhouse.chdb:chdb-arrowmodule using Arrow Java 19.0.0 and the native Arrow C Data/C Stream ABI. This PR pins the published chdb-core v26.11.0-rc.3 across four platform packages, with checksums from the release and a header copied from its tarball. The binding's development version becomes1.0.0-rc.3-SNAPSHOT; this PR does not publish it. Existing JDBCResultSet/RowBinary semantics and the ordinary JDBC dependency tree are unchanged.A1–A6
VectorSchemaRootcurrent batch as an in-memory SQL source with a closable registration and opaquearrowstream('name')fragment.ArrowReaderas a lazily consumed source; the registration owns the reader and holds its C Stream until scans finish before unregistering.ArrowQuerypulls typed batches incrementally, with partial-consumption close, cooperative cancel, and connection/native handle leases.chdb_insert_arrow_stream; source columns must have distinct names and the destination identifier is validated and quoted. No implicit table creation or transaction guarantee.Verification
v26.11.0-rc.3macOS arm64 asset SHA-2567b0ea049efcf75a851e744c7d74325d7a9b143227f9b72097e5df252564e3d58; local extraction, header, and staged JNI pair verified withoutskipEngineVersionCheck.psfor an RSS test; the approved rerun passed.LicenseInventoryITmatches 1081 components generated from the rc.3 binary; the only difference from rc.2 is removal ofgdeflate(Apache), with no copyleft change.build.ymlruns the native Arrow suite on Linux x86_64/aarch64 and macOS arm64/x86_64 × JDK 11/17/21/25, with no rc.2 skip or engine-version override. A Maven failure fails its platform job and Arrow reports/crash logs are archived. These remote combinations remain to be observed on this PR.Limits and follow-up
The disabled insert-conversion regression is not counted as passed. It was reproduced on the published rc.3 macOS arm64 library: an Arrow UTF-8
id="not-an-integer"inserted intoInt64reports one row written and stores0, while equivalent ordinary SQL rejects it withCANNOT_PARSE_TEXT. Re-enable the regression after core resolves that behavior; do not silently coerce in Java. A2 is one-shot; A4 cancellation waits for an in-flight fetch; A5 targets existing tables. Production use and a subsequent Java RC still require review of this PR's four-platform results and the normal publishing gates.Note
Add
chdb-arrowmodule and bump engine baseline to v26.11.0-rc.3chdb-arrowmodule:ArrowSessionprovides materialized and incremental Arrow queries, registered Arrow sources, and synchronous Arrow inserts, withArrowOptionscontrolling type-mapping flags (default maps String as string)ChdbConnectiontracks Arrow queries and cleanup hooks;close()closes tracked queries and runs hooks before closing statementsChdbConnection.arrowStreamOpenNativeholds the statement slot for the stream's lifetime, so an unclosed Arrow stream blocks other statementsMacroscope summarized 6e9f393.