Report peak memory per process instead of a summed total - #6297
Open
ondrejmirtes wants to merge 1 commit into
Open
Report peak memory per process instead of a summed total#6297ondrejmirtes wants to merge 1 commit into
ondrejmirtes wants to merge 1 commit into
Conversation
The printed number was the sum of every worker's peak plus a snapshot of
the main process's current usage, which described a moment that never
happens: workers do not peak at the same time, and the main process's own
peak comes later still, while it collects the workers' results and saves
the result cache. The snapshot understated it threefold (136 MB recorded
against a 405 MB peak on a self-analysis run).
Fork made it worse: memory_get_peak_usage() carries over into a
pcntl_fork()-ed child, so every worker reported the main process's peak
before allocating anything of its own. Summed across workers that
multiplied a spike the main process took before forking - loading the
result cache on an incremental run - by the number of workers, and one
project displayed 29 GB while no process ever exceeded 3.2 GB.
Report what each process actually reached instead: this process's peak,
read at the very end, and the heaviest worker's - which is also the
number memory_limit applies to, since it is a per-process limit. The
forked child restarts its peak tracking so the figure is its own.
Peak memory: 3.24 GB (main process), 1.41 GB (largest of 10 forked workers)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014fMtimQECyyBNKTmFddr8L
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Verbose runs printed
Used memory:— a number that never corresponded to anything the operating system could confirm. This replaces it with two figures that can be checked:Single-process runs print
Peak memory: 401.02 MB.Why the old number was fiction
It was
array_sum()of every worker's peak, plus a snapshot of the main process's current usage taken the moment the first worker exited. Workers never peak at the same instant, so the sum described a moment that never happened. On top of that, two concrete defects:The main process was understated. We recorded current usage, sampled early, while the main process's real peak comes much later — when it aggregates worker results and writes the result cache. On PHPStan's own self-analysis we recorded 136 MB for a process that actually peaked at 405 MB.
memory_get_peak_usage()is inherited acrosspcntl_fork(). A child reports its parent's peak before allocating anything:Summed across workers, that multiplied the main process's pre-fork spike — on an incremental run, the loaded result cache — by the number of workers.
What the OS saw, before this change
Two real projects, PHP 8.5, 10 workers. "Largest process" is the kernel's
maximum resident set size; the sum column is RSS polled across the whole process tree.It was wrong with spawning too. Lorem spawned: 7.55 GB printed against a largest process of 0.98 GB — 7.7×. Dolor spawned: 11.46 GB printed, 2.89 GB largest, and 18% below the sum it was nominally computing (because the main process's contribution was that early snapshot).
Forking made it far worse. Same project, same work — the only difference is that the main process had the result cache loaded when it forked: 29.21 GB printed against a 3.14 GB largest process, 9.3×, and 1.75× even the sum of every process's peak. Nothing on that machine ever held 29 GB.
After this change
Where the main process is the largest one, the printed figure matches the kernel's within 1–5%. Worker figures land within 3–13% in spawn mode. Both are per-process, which is also how
memory_limitapplies — so if a run dies at its limit, these are the numbers that explain why.What these numbers are not
They are not the memory the run costs the machine. Sampling machine-wide physical memory during these same runs gives ~5.6 GB (Dolor fork, cold) and ~10.3 GB (Dolor fork, warm) — more than any single process, far less than the sum, because forked workers share most of their pages with the parent.
No number derived from
memory_get_*can express that: when a forked worker writes to an inherited page, copy-on-write allocates real memory while ZendMM sees nothing. The one honest figure for fork mode is the worker heap number above, which includes memory the worker inherited and still holds — an upper bound on its private footprint (2.19 GB printed against 1.48 GB resident in the warm-cache row).A machine-level footprint would need OS counters —
Pssfrom/proc/<pid>/smaps_rollupon Linux,phys_footprinton macOS — and is deliberately left for separate work.The change
ForkedProcesscallsmemory_reset_peak_usage()in the child right after the fork, so a worker reports its own high-water mark instead of its parent's.ParallelAnalyserstops folding the main process into the total;peakMemoryUsageBytesnow means "the heaviest worker's peak", and single-process paths report0.InceptionResultreads this process's peak at the very end — after the result cache is written — and prints both figures with the worker count and mechanism.