[Variant] Match Spark physical Parquet semantics for projected Variant columns

[peterxcli]

Problem

Arrow-rs and Spark can derive different Arrow types and name matches from the same Parquet file. For projected Variant storage this can change the reconstructed value, reject a file Spark reads, or select the wrong physical column.

Parent: #5546
Depends on: #5549
Cleanup trackers: #5474, #5477
Resolved prerequisite: #5495 (merged in #5602)

Solution

At the projected, explicitly marked Variant boundary only:

• Prevent advisory ARROW:schema types from overriding Spark's physical Parquet interpretation.
• Preserve Parquet ENUM as a Variant string while leaving raw BINARY as binary.
• Match Spark's physical decimal, date/timestamp, fixed binary/list, and encoded child behavior needed by whole-value reconstruction.
• Preserve row-group metadata and page/offset indexes when rebuilding reader metadata.
• Keep encrypted affected scans on Spark when reader metadata cannot be reconstructed safely.
• Reuse the shared Unicode name matching and duplicate detection landed in feat: support unicode case sensitive field names for reading parquet #5602. Do not port feat: project Spark 4 VARIANT columns in native Parquet scans [Splitting] #5407's Variant-only Unicode fail-closed guard.
• Handle remaining field-ID shadowing while preserving Spark's ID-before-name precedence. This is separate from the generic Unicode matching fixed by feat: support unicode case sensitive field names for reading parquet #5602.

The JVM Variant scan gate remains closed in this task.

Tests

Use generated Parquet inputs rather than committed fixtures for:

• encoded storage children and supported unsigned widths;
• timestamp, fixed binary, and fixed-size-list storage;
• positive and negative wide DECIMAL values;
• ENUM versus UTF8 versus raw BINARY;
• advisory Arrow-schema hints;
• field-ID shadowing and integration with the shared Unicode name matcher;
• encrypted fallback; and
• pruning that does not activate compatibility work for an unread Variant column.

Each compatibility branch must link its upstream owner and the corresponding cleanup tracker.

[peterxcli]

take