[spark] Support source table fileio for blob descriptors on spark - #9598
[spark] Support source table fileio for blob descriptors on spark#9598Stefanietry wants to merge 1 commit into
Conversation
6bf62c7 to
d1be1a2
Compare
| metadataColumns: Seq[PaimonMetadataColumn] = Seq.empty, | ||
| blobAsDescriptor: Boolean) | ||
| blobAsDescriptor: Boolean, | ||
| uriReaderFactory: UriReaderFactory = null) |
There was a problem hiding this comment.
[P1] Propagate the descriptor reader through every Spark read path
This new dependency defaults to null, and PaimonMicroBatchStream#createReaderFactory still constructs PaimonPartitionReaderFactory with only readBuilder/blobAsDescriptor. Every Structured Streaming read therefore takes the uriReaderFactory == null branch and keeps the BlobRef reader built from the target table FileIO, so descriptors that require blob-descriptor-source-table still fail on toData(). The postpone.merge-on-read path has the same gap: SortedBucketMergeIterator converts currentReader.next() directly and MergePlan carries only blobAsDescriptor. Please build/pass the source UriReaderFactory through both paths (and add streaming/postponed-read regressions) instead of silently falling back to null.
| field => PaimonMetadataColumn.get(field.name, SparkTypeUtils.toSparkPartitionType(table))) | ||
| PaimonBatch(inputPartitions, readBuilder, coreOptions.blobAsDescriptor(), metadataColumns) | ||
| val uriReaderFactory = table match { | ||
| case fileStoreTable: FileStoreTable => BlobDescriptorReaderFactory.create(fileStoreTable) |
There was a problem hiding this comment.
[P1] Do not resolve the source table when descriptor output is requested
BlobDescriptorReaderFactory.create is called for every FileStoreTable before checking coreOptions.blobAsDescriptor(). With blob-as-descriptor=true the partition reader deliberately never uses this factory, but create may still open the configured catalog and load blob-descriptor-source-table. A descriptor-only query (or even a projection with no BLOB column) now fails if that source table is unavailable, although returning the serialized descriptor requires no source data access. Please gate factory creation on !blobAsDescriptor (and preferably on the projected read type containing a BLOB) so unused source-table resolution cannot break scans.
2 participants
Purpose
Purpose: blob-descriptor-field stores blob values as serialized BlobDescriptor bytes inline in data files. In Spark, reading such fields as normal binary data should return the actual blob bytes, while blob-as-descriptor=true should return descriptor bytes.
Currently, spark may fail to resolve the actual blob data because the read path does not always attach the required descriptor reader to the reconstructed blob value.
Linked issue: #9582
Tests