[GH-3124] Use native spatial types on Spark 4.2

[jiayuasu]

Did you read the Contributor Guide?

Yes, I read the Contributor Rules and Contributor Development Guide.

Is this PR related to a ticket?

Closes #3124.

What changes were proposed in this PR?

Sedona's UDT columns cannot be passed directly to Spark 4.2's native spatial functions. This adds a Spark 4.2 build that exposes native GEOMETRY/GEOGRAPHY columns, while older Spark builds retain their existing UDT contract.

Spark owns ST_AsBinary, ST_GeomFromWKB, ST_GeogFromWKB, ST_SRID, and ST_SetSRID on 4.2. Sedona preserves those registrations and delegates its existing wrappers to Spark, including the geography constructor's native 4326 default.

The native build adapts Sedona's computational expressions at their inputs and outputs. It preserves aggregate and generator identity, nested spatial results, nulls, SRIDs, dimensional WKB, and public expression names. The same boundary applies to file formats, RDD adapters, box casts, spatial filter pushdown, and SQL clustering. Adjacent Sedona calls avoid intermediate serialization. Python uses PySpark spatial types with explicit Shapely conversion and updated GeoPandas, GeoArrow, and vectorized UDF boundaries. Existing UDT-backed data remains readable and can be converted through EWKB.

The bridge reads and writes Spark's SRID-prefixed ISO WKB directly. Output validation happens while writing JTS coordinates; input validation reads the native buffer without constructing Spark geometry objects. JTS receives the correct SRID factory on its first parse, and lazy geography values own their payload after a source row is reused. This removes redundant WKB rewrites and geometry copies while preserving Spark's dimensional, coordinate, ring, geography-bound, and SRID checks. Spark's own function and encoder implementations remain unchanged.

The migration guide documents signatures, SRID validation, native Python values, and fixed versus mixed SRID persistence. Spark 4.2.0's default columnar cache still rejects native spatial columns; DBSCAN caches vertex IDs for its internal graph cache. General native DataFrame caching remains a Spark limitation. Mixed-dimensional collections follow Spark's rejection contract.

How was this patch tested?

Latest Java 17 verification:

• mvn -B -ntp -Dspark=4.2 -Dscala=2.13 -pl spark/spark-4.2 -am -Dmaven.test.skip=true -DskipTests -Dmaven.javadoc.skip=true install — passed.
• mvn -B -ntp -Dspark=4.2 -Dscala=2.13 -pl spark/spark-4.2 test — 62 passed across 10 suites.
• mvn -B -ntp -Dspark=4.2 -Dscala=2.13 -pl spark/common test -Dsuites=org.apache.sedona.sql.UDF.NativeFunctionRegistrationTest -Dtest=NativeFunctionRegistrationTest -DfailIfNoTests=false -Dsurefire.failIfNoSpecifiedTests=false — 9 passed; one legacy-only case canceled as intended.
• The module includes 20 new reader/writer/validator tests: XY/Z/M/ZM and empties, multipart promotion, mixed byte order, precise Spark validation-error parity, geography bounds, slices/off-heap memory and ownership, truncations, and 500 deterministic payload mutations.
• Spotless, git diff --check, and independent source review passed. The full repository suite was not run locally.

JMH 1.37 geometry SerDe rerun: Java 17.0.13, ARM64, one thread, two forks, 3 x 500 ms warmup and 5 x 500 ms measurement per fork, fixed 1 GiB heap, ActiveProcessorCount=2, GC profiler. Same input geometries (XY, SRID 4326) and benchmark sources as the previous run. Times are mean microseconds per operation; these measure JTS-to-representation boundaries, not complete Spark queries.

Geometry	Operation	Previous bridge	Optimized bridge	Spark 4.0.2 UDT
Point	Encode	0.094	0.025	0.005
Point	Decode	0.102	0.075	0.012
LineString, 1,000 vertices	Encode	37.768	2.443	0.989
LineString, 1,000 vertices	Decode	42.049	21.335	3.015
MultiPoint, 1,000 points	Encode	99.180	18.466	4.951
MultiPoint, 1,000 points	Decode	119.067	80.180	10.177

Encoding improved 3.7-15.5x; line decoding improved about 2x. MultiPoint decoding has wide confidence intervals overlapping the earlier baseline, so its lower mean is not a reliable speedup claim. Large-geometry allocation fell by approximately 64-91%. Unchanged custom-serializer controls remained close to their earlier results. Spark 4.0.2 UDT is still faster, especially at decoding. The 4.0 control uses the existing Sedona snapshot UDT wrapper with the same current common serializer, rather than a fresh full Spark 4.0 build.

Earlier branch validation (before the latest native codec optimization): Spark 3.5.8 registration/GeoStats (19 passed) and GeoParquet (6 passed); the original Python spatial-index failure (1 passed); Spark 4.1.2/3.5.4 UDT registration and file-format checks; Python Classic native/GeoArrow (30 passed) and actual local Spark Connect (22 passed); Spark 4.1 UDT Parquet migration to 4.2 with geography collections and XYZM geometry. Older Python GeoArrow (8 passed) and vectorized UDF checks (7 passed) also passed. Four broader GeoPandas dimension failures reproduced unchanged against the released 1.9.1 artifact. Those older/Python checks were not rerun for the native-only codec implementation.

Did this PR include necessary documentation updates?

Yes: a Spark 4.2 migration guide, Python conversion examples, Maven coordinates, and navigation links.