Skip to content

Add optional Arrow Java integration pinned to chdb-core v26.11.0-rc.3 - #37

Closed
ShawnChen-Sirius wants to merge 2 commits into
chdb-io:mainfrom
ShawnChen-Sirius:codex/arrow-java
Closed

ShawnChen-Sirius wants to merge 2 commits into
chdb-io:mainfrom
ShawnChen-Sirius:codex/arrow-java

Conversation

@ShawnChen-Sirius

@ShawnChen-Sirius ShawnChen-Sirius commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Add the optional com.clickhouse.chdb:chdb-arrow module using Arrow Java 19.0.0 and the native Arrow C Data/C Stream ABI. This PR pins the published chdb-core v26.11.0-rc.3 across four platform packages, with checksums from the release and a header copied from its tarball. The binding's development version becomes 1.0.0-rc.3-SNAPSHOT; this PR does not publish it. Existing JDBC ResultSet/RowBinary semantics and the ordinary JDBC dependency tree are unchanged.

A1–A6

  • A1: caller-owned VectorSchemaRoot current batch as an in-memory SQL source with a closable registration and opaque arrowstream('name') fragment.
  • A2: one-shot Apache Arrow ArrowReader as a lazily consumed source; the registration owns the reader and holds its C Stream until scans finish before unregistering.
  • A3: core-materialized Arrow C Stream result exposed as a typed Arrow Java reader, retaining schema for zero rows.
  • A4: ArrowQuery pulls typed batches incrementally, with partial-consumption close, cooperative cancel, and connection/native handle leases.
  • A5: synchronous insert into an existing table from a batch or reader through chdb_insert_arrow_stream; source columns must have distinct names and the destination identifier is validated and quoted. No implicit table creation or transaction guarantee.
  • A6: pinned core/JNI ABI and platform manifests, Java 11 optional module, explicit Arrow output type choices, ownership rules, and no implicit IPC or unconditional zero-copy claim.

Verification

  • Published v26.11.0-rc.3 macOS arm64 asset SHA-256 7b0ea049efcf75a851e744c7d74325d7a9b143227f9b72097e5df252564e3d58; local extraction, header, and staged JNI pair verified without skipEngineVersionCheck.
  • On that published pair, JDK 11 and 21 each pass 21/22 Arrow cases; one explicitly disabled regression documents a core String → Int64 silent conversion (upstream issue to be filed separately). JDBC unit tests: 178; the complete JDBC integration suite on JDK 11/macOS arm64: 177. The first sandboxed run could not execute ps for an RSS test; the approved rerun passed.
  • LicenseInventoryIT matches 1081 components generated from the rc.3 binary; the only difference from rc.2 is removal of gdeflate (Apache), with no copyleft change.
  • PR build.yml runs the native Arrow suite on Linux x86_64/aarch64 and macOS arm64/x86_64 × JDK 11/17/21/25, with no rc.2 skip or engine-version override. A Maven failure fails its platform job and Arrow reports/crash logs are archived. These remote combinations remain to be observed on this PR.

Limits and follow-up

The disabled insert-conversion regression is not counted as passed. It was reproduced on the published rc.3 macOS arm64 library: an Arrow UTF-8 id="not-an-integer" inserted into Int64 reports one row written and stores 0, while equivalent ordinary SQL rejects it with CANNOT_PARSE_TEXT. Re-enable the regression after core resolves that behavior; do not silently coerce in Java. A2 is one-shot; A4 cancellation waits for an in-flight fetch; A5 targets existing tables. Production use and a subsequent Java RC still require review of this PR's four-platform results and the normal publishing gates.

Note

Add chdb-arrow module and bump engine baseline to v26.11.0-rc.3

  • Adds an optional chdb-arrow module: ArrowSession provides materialized and incremental Arrow queries, registered Arrow sources, and synchronous Arrow inserts, with ArrowOptions controlling type-mapping flags (default maps String as string)
  • Implements the native side in chdb_jni.cpp: JNI entrypoints for materialized queries, stream open/fetch/cancel/close, registration, and insertion, backed by Arrow stream handles (kind 5)
  • ChdbConnection tracks Arrow queries and cleanup hooks; close() closes tracked queries and runs hooks before closing statements
  • Pins the engine to 26.11.0-rc.3 (SHA-256 values and license inventory updated) and bumps the JNI ABI from 2 to 3; CI runs the Arrow integration tests per Java version
  • Risk: JNI ABI bump to 3 means older native shims fail to load; ChdbConnection.arrowStreamOpenNative holds the statement slot for the stream's lifetime, so an unclosed Arrow stream blocks other statements

Macroscope summarized 6e9f393.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant