POC: Parallel LPT buckets

[webard]

Created with the help of Claude.

A proof of concept for testing, not a merge candidate. No tests yet, an opt-in experimental flag, and one unresolved trade-off (memory) that would have to be settled before this could ship. Opening it to put the approach and the numbers in front of people who can try it on their own codebases.

Follows up on #8494, and specifically @samsonasik's comment there: #8494 (comment)

All of it behind --lpt, opt-in. Without the flag the default path is untouched.

scheduler	wall	peak RSS
current	152.2 s	751 MB
LPT, static buckets	127.8 s	938 MB
LPT + stealing	113.4 / 122.1 s	902 MB

jobSize sweep, LPT + stealing, 14 workers:

jobSize	wall	diffs
1	98.8 s	3 338
2	90.7 / 91.4 s	3 338
4	96.7 / 99.2 s	3 338
8	104.6 s	3 338
16	113.4 s	3 338
50	118.1 s	3 338
150	127.5 s	3 237 — timed out

Best current 152.2 s → best --lpt 90.7 s, i.e. -40 %. At identical worker count and jobSize, -25 %.

Every run produces the same 3 338 diffs (sha over the file-sorted diff list) and the same 2 errors, except jobSize: 150.

vendor/bin/rector process --dry-run --clear-cache --no-progress-bar

vendor/bin/rector process --dry-run --clear-cache --no-progress-bar --lpt

[samsonasik]

Thank you @webard, I tested locally main vs your pr branch and your pr branch looks faster 👍

main branch: 14.645 seconds.

➜ (main) time bin/rector --no-diffs --dry-run --clear-cache --no-progress-bar       

129.32s user 12.90s system 971% cpu 14.645 total

poc/parallel-lpt-buckets branch with use of --lpt: 11.455 seconds.

➜ (poc/parallel-lpt-buckets) time bin/rector --no-diffs --dry-run --clear-cache --no-progress-bar --lpt 
                                                                                                                         
108.58s user 10.59s system 1040% cpu 11.455 total