Conversation
zghong
requested review from
924060929,
CalvinKirs,
dataroaring,
englefly,
gavinchou,
morningman,
morrySnow,
starocean999,
xiaokang and
yiguolei
as code owners
August 5, 2026 07:28
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Author
|
run buildall |
Contributor
TPC-H: Total hot run time: 29153 ms |
Contributor
TPC-DS: Total hot run time: 166066 ms |
Contributor
ClickBench: Total hot run time: 23.98 s |
…ion_hash_type identity
…ash_type # Conflicts: # be/src/exec/partitioner/partitioner.h # fe/fe-core/src/main/java/org/apache/doris/planner/HashDistributionPruner.java # fe/fe-core/src/test/java/org/apache/doris/planner/HashDistributionPrunerTest.java
Carry the effective storage hash type through FE-planned and BE-native local bucket exchanges. Add fragment protocol metadata and execution-version gating so older backends cannot silently execute IDENTITY layouts as CRC32.
Author
|
run buildall |
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
FE Regression Coverage ReportIncrement line coverage |
…egacy values, and local-exchange hash
Author
|
run buildall |
Contributor
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
TPC-H: Total hot run time: 16901 ms |
Contributor
TPC-DS: Total hot run time: 81949 ms |
Contributor
ClickBench: Total hot run time: 14.7 s |
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
|
[P1] Preserve the probe-side IDENTITY layout through broadcast joins At commit A reduced plan illustrating the issue is: The broadcast join reports I reproduced this with a focused FE unit test: construct a real broadcast On a multi-BE layout, CRC32 recomputation can select buckets absent from the current BE's local bucket map. The local exchanger's row-count check then returns The BE-native local-exchange path also derives the fragment hash type from the plan root, so the same loss can leave its bucket partitioner at CRC32. Please derive the hash type from the distribution actually preserved by each operator, with broadcast joins inheriting the probe side, and add a broadcast-join → bucket-join regression covering both FE-planned and BE-native local exchange. |
Contributor
|
[P1] Normalize storage hash metadata when converting to EXECUTION_BUCKETED ChildrenPropertiesRegulator.calAnotherSideRequired() unconditionally copies This can produce an However, A reduced problematic property combination is: Both sides physically use the same execution shuffle algorithm. The regulator's both-EXECUTION_BUCKETED branch checks key order, but the new DistributionSpecHash.merge() assertion rejects the different retained storage labels. I reproduced the failure in a focused FE unit test by constructing these two distribution specs, passing them to the real This is a property-derivation unit reproduction with directly constructed child properties and a mocked join type, not an end-to-end SQL reproduction. The producer of the inconsistent properties is the downgrade/helper path above. Please normalize the storage hash metadata when transitioning to EXECUTION_BUCKETED, and define hash-type equality according to the relevant distribution kind. Merely removing the merge assertion would leave the metadata inconsistent with execution. A regression should combine an identity-origin execution-shuffled subplan with a CRC32-origin execution-shuffled subplan. |
Broadcast joins do not repartition their probe input, so derive the output storage hash type from the probe child instead of combining it with the replicated build child. This keeps IDENTITY metadata intact for downstream FE-planned and BE-native bucket local exchanges, with focused unit and regression coverage for broadcast-join-to-bucket-join plans.
EXECUTION_BUCKETED is produced by the ordinary execution exchange and no longer follows a table's storage hash algorithm. Normalize its hash metadata to CRC32 at DistributionSpecHash construction so downgraded IDENTITY plans can merge with other execution-shuffled inputs without reporting a false storage-layout conflict.
…ash_type # Conflicts: # be/src/agent/be_exec_version_manager.cpp # be/src/agent/be_exec_version_manager.h # be/test/exec/sink/tablet_sink_hash_partitioner_test.cpp # fe/fe-common/src/main/java/org/apache/doris/common/Config.java
The merged master branch rejects JUnit 3/4 APIs and already migrated overlapping tests to JUnit 5. Update the distribution hash tests and newly merged assertions to use JUnit 5 so FE test compilation and targeted tests pass.
Author
|
run buildall |
Contributor
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
TPC-H: Total hot run time: 16978 ms |
Contributor
TPC-DS: Total hot run time: 84138 ms |
Contributor
ClickBench: Total hot run time: 14.9 s |
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
TPC-H: Total hot run time: 17044 ms |
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
TPC-DS: Total hot run time: 83049 ms |
Contributor
ClickBench: Total hot run time: 14.81 s |
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
Author
|
run buildall |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: close #66231
Problem Summary:
Hash Bucketing previously supported only the CRC32 hash function to map a row to its bucket. There was no way for users to choose a different bucketing scheme, which prevents scenarios (e.g. bucket pruning / colocation with external systems) that rely on a plain identity mapping of the bucket key, as shown in #66231.
This PR introduces a table-level
distribution_hash_typeproperty for Hash Bucketing tables, supporting two values:crc32(default, keeps existing behavior), meanscrc32(key) % bucket_num.identity(map rows to buckets by the raw key value), meanskey % bucket_num.Main changes:
distribution_hash_typeproperty inPropertyAnalyzer; carry the hash type throughHashDistributionInfo,HashDistributionDesc,ColocateGroupSchema,DataPartition,DistributionSpecHashand related create-table / colocate paths; a newFeMetaVersionis added for metadata persistence.ADD PARTITIONnow inherits the table hash type.HashDistributionPruner,PruneOlapScanTablet) still works correctly for identity buckets.TDistributionHashTypeis added to thrift and threaded through the exchange / local-exchange sink operators and the tablet-sink partitioner so that the backend uses the same bucketing function as the FE plan, keeping FE/BE bucketing consistent. An identity partitioner is implemented.TDistributionHashTypeand propagate the field inDescriptors,Partitions,PlanNodesandTypes.Before this PR: only CRC32 bucketing was available.
After this PR: users can create Hash Bucketing tables with
"distribution_hash_type" = "identity", and FE plan optimization plus BE data sink produce consistent bucketing results. And more hash functions can be extended in the future.Release note
Support a new table property
distribution_hash_typefor Hash Bucketing tables. It acceptscrc32(default) andidentitynow, allowing users to choose the hash function used to map rows to buckets.Check List (For Author)
Test
Behavior changed:
distribution_hash_typeproperty is added; but default behavior (crc32) is unchanged.Does this need documentation?
Check List (For Reviewer who merge this PR)