[improvement](be) Prune rowsets by TSO before row binlog scans - #68050
Merged
morningman merged 1 commit intoSep 16, 2026
Merged
Conversation
### What problem does this PR solve? Issue Number: None Related PR: apache#68012 Problem Summary: Bounded ROW binlog queries initialize readers and open segment footers for historical rowsets before TSO predicates reject them. Prune the captured version-consistent read source using inclusive rowset TSO metadata and the query's half-open window. Preserve unknown metadata, overlapping compaction output, and separately captured delete predicates. Skip scanners when only empty rowsets remain, and expose pruning counters in the profile. Port commit 93a3cec to master without conflicts; the stable patch ID is unchanged. ### Release note Reduce reader initialization and segment footer IO for bounded ROW binlog queries. ### Check List (For Author) - Test: Manual test (port validation) - clang-format 16.0.6 repository check and BE header hygiene passed. - Source whitespace checks and stable patch-ID comparison passed. - Existing unit tests and generated regression fixtures are carried over. - Build, unit/regression tests, and clang-tidy were not rerun on this base. - Original PR records ASAN build, 8 unit tests, regression and profile validation. - Behavior changed: No (query results unchanged; profile counters added) - Does this need documentation: No
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
/review |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Static review summary for exact head 6caf5fda1ed033214cf0642bc805904e7081da6c
Verdict: no actionable findings; I would approve this change.
- Goal and scope: the patch is narrowly focused on pruning captured ROW-binlog rowsets by their inclusive commit-TSO envelope before reader cloning and segment-footer I/O. It also adds targeted profile counters and focused unit/regression coverage.
- Correctness and lifecycle: the rowset disjointness test correctly coarse-filters the downstream half-open
[start_tso, end_tso)predicate. Unknown legacy endpoints are retained. Pruning preserves separately captured delete predicates/bitmap and materialized before images; empty/bootstrap-only sources skip scanner construction and reach EOS correctly, while mixed-tablet scans retain their remaining scanners. - Compatibility and special paths: no configuration, FE-BE protocol, persisted format, or write/transaction behavior changes. Local/cloud publish paths, protobuf conversion, and normal/ordered compaction preserve conservative metadata envelopes. NONE, APPEND_ONLY, MIN_DELTA, and DETAIL use the same TSO coordinate and bounds.
- Concurrency, performance, and observability: pruning is synchronous before scanner workers start and introduces no shared-state or lock-order change. The added work is linear in captured rowsets and can avoid reader clones/footer I/O; the timer and rowset/segment/tablet counters make the optimization visible.
- Tests/results: the unit tests cover exact and one-sided boundaries, compacted overlaps, unknown metadata, captured delete state, and all-pruned EOS/bootstrap behavior. The regression covers DETAIL/MIN_DELTA before images and deletes, one-sided/empty windows, snapshots, and local compaction with deterministic ordered output. No additional user review focus was provided.
Validation note: this was a static-only review under the supplied contract; I did not run builds or tests. The PR reports validation for the original patch, but current-base execution was not independently rerun in this review.
Contributor
Author
|
run buildall |
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
TPC-H: Total hot run time: 16866 ms |
Contributor
TPC-DS: Total hot run time: 81681 ms |
Contributor
ClickBench: Total hot run time: 14.67 s |
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
morningman
approved these changes
Sep 16, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: None
Related PR: #68012 (merged into
branch-incremental-computation); this PR ports the same change tomaster.Problem Summary:
Bounded ROW binlog queries initialize rowset readers and open segment footers before their TSO predicates reject historical data. A narrow time window can therefore pay initialization costs proportional to the retained history.
Prune the captured read source before cloning readers or creating scanners. For query
[100, 200), a rowset with inclusive commit TSO range[10, 99]can be skipped using its metadata alone.RowsetTsoPruneTime,RowsetsPrunedByTso,SegmentsPrunedByTso, andTabletsPrunedByTsoto the query profile.The filtering uses
std::erase_ifon the existing rowset vector andstd::all_offor the empty-tablet check. It adds no storage-format or protocol changes.Release note
Reduce reader initialization and segment footer IO for bounded ROW binlog queries.
Check List (For Author)
Validation:
Current master port (base
060c7dbe81758fb0d673567fb361234d06b9a1b1):93a3ceced9c4518d527c2b2d58ee328bb5960d03without conflicts. Stable patch IDs match the source PR exactly.build-support/check-format.shpassed with clang-format 16.0.6.build-support/check-build-hygiene.shpassed..outis preserved unchanged, including its trailing blank line.Historical validation reported in #68012 for the original master-based implementation (
cb73d23c873, based on96d0ac68e84):build.sh --be --fe -j 48.OlapScanOperatorTsoPruningTestandOlapScanOperatorBinlogPushDownTestpassed under ASAN. Coverage includes half-open boundaries, single/no bounds, unknown metadata, overlapping compaction output, delete predicates, and empty-source EOS.test_binlog_rowset_tso_pruningandtest_binlog_changes_syntaxpassed. The new output was generated by the regression runner and then checked in a second run. Before-images, deletes, empty windows, and compaction results were verified.Manual profile check: create a one-bucket UNIQUE KEY table with ROW binlog, historical values, and automatic compaction disabled; insert two rows in two separate transactions. Enable profiling and query
@incr('startTimestamp'='2099-01-01 00:00:00', 'incrementType'='MIN_DELTA'). The empty result reported:This verifies that scanner initialization is skipped. No production latency improvement is claimed; rowset metadata synchronization and version capture still occur.
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)