[AURON #2507] Optimize partial WindowGroupLimit execution - #2508
[AURON #2507] Optimize partial WindowGroupLimit execution#2508lyne7-sc wants to merge 4 commits into
Conversation
slfan1989
left a comment
There was a problem hiding this comment.
Thanks for this optimization.
The specialized processor correctly maintains partition, ordering, and ranking state across batches, and the range-based filtering is well scoped.
The tests cover row_number, rank, and dense_rank, including peer groups spanning batch boundaries, and the CI checks are green.
Two minor documentation nits:
-
The benchmark table appears to be out of sync with the checked-in benchmark names. It lists
window_group_limit_row_number_single_partitionandwindow_group_limit_row_number_all_selected, while the current benchmark file defines the*_slicecases instead. -
Please include the actual tool name in the AI disclosure, for example
Generated-by: OpenAI Codex (GPT-5)if applicable.
No blocking code concerns otherwise. LGTM!
Which issue does this PR close?
Closes #2507
Rationale for this change
Partial
WindowGroupLimitExeccurrently materializes and filters a ranking column that is not included in its output. It also always uses Arrow filtering even when the selected rows form a contiguous range.What changes are included in this PR?
row_number,rank, anddense_rank.RecordBatch::slicefor a single contiguous range.Are there any user-facing changes?
No. This is an internal performance optimization.
How was this patch tested?
Passed unit tests.
Benchmarks
The benchmark measures WindowGroupLimit execution over 100,000 input rows. Input sorting is excluded from the measurement.
window_group_limit_row_numberwindow_group_limit_rankwindow_group_limit_dense_rankwindow_group_limit_row_number_single_partitionwindow_group_limit_row_number_all_selectedWas this patch authored or co-authored using generative AI tooling?
If yes, include:
Generated-by: GPT-5ASF guidance: https://www.apache.org/legal/generative-tooling.html