Skip to content

[AURON #2507] Optimize partial WindowGroupLimit execution - #2508

Open
lyne7-sc wants to merge 4 commits into
apache:masterfrom
lyne7-sc:perf/window-group-limit
Open

[AURON #2507] Optimize partial WindowGroupLimit execution#2508
lyne7-sc wants to merge 4 commits into
apache:masterfrom
lyne7-sc:perf/window-group-limit

Conversation

@lyne7-sc

@lyne7-sc lyne7-sc commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Closes #2507

Rationale for this change

Partial WindowGroupLimitExec currently materializes and filters a ranking column that is not included in its output. It also always uses Arrow filtering even when the selected rows form a contiguous range.

What changes are included in this PR?

  • Add a specialized partial group-limit processor for row_number, rank, and dense_rank.
  • Track partition, ordering, and ranking state across input batches.
  • Collect adjacent selected rows as ranges instead of building the selection bitmap row by row.
  • Use zero-copy RecordBatch::slice for a single contiguous range.
  • Build the Boolean selection in blocks for multiple ranges.
  • Add correctness tests and operator-level benchmarks for contiguous and multi-range selections.

Are there any user-facing changes?

No. This is an internal performance optimization.

How was this patch tested?

Passed unit tests.

Benchmarks

The benchmark measures WindowGroupLimit execution over 100,000 input rows. Input sorting is excluded from the measurement.

Benchmark Master This PR Improvement
window_group_limit_row_number 17.558 ms 13.538 ms 22.9%
window_group_limit_rank 32.092 ms 27.360 ms 14.7%
window_group_limit_dense_rank 32.683 ms 28.133 ms 13.9%
window_group_limit_row_number_single_partition 17.612 ms 13.175 ms 25.2%
window_group_limit_row_number_all_selected 17.436 ms 14.519 ms 16.7%

Was this patch authored or co-authored using generative AI tooling?

  • Yes
  • No

If yes, include: Generated-by: GPT-5

ASF guidance: https://www.apache.org/legal/generative-tooling.html

@github-actions github-actions Bot added the native label Sep 4, 2026
@slfan1989 slfan1989 self-assigned this Sep 7, 2026

@slfan1989 slfan1989 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this optimization.

The specialized processor correctly maintains partition, ordering, and ranking state across batches, and the range-based filtering is well scoped.

The tests cover row_number, rank, and dense_rank, including peer groups spanning batch boundaries, and the CI checks are green.

Two minor documentation nits:

  • The benchmark table appears to be out of sync with the checked-in benchmark names. It lists window_group_limit_row_number_single_partition and window_group_limit_row_number_all_selected, while the current benchmark file defines the *_slice cases instead.

  • Please include the actual tool name in the AI disclosure, for example Generated-by: OpenAI Codex (GPT-5) if applicable.

No blocking code concerns otherwise. LGTM!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Performance] Optimize partial WindowGroupLimit execution

2 participants