Skip to content

Report peak memory per process instead of a summed total - #6297

Open
ondrejmirtes wants to merge 1 commit into
2.2.xfrom
fix-fork-memory-reporting
Open

Report peak memory per process instead of a summed total#6297
ondrejmirtes wants to merge 1 commit into
2.2.xfrom
fix-fork-memory-reporting

Conversation

@ondrejmirtes

Copy link
Copy Markdown
Member

Verbose runs printed Used memory: — a number that never corresponded to anything the operating system could confirm. This replaces it with two figures that can be checked:

Peak memory: 3.07 GB (main process), 1.18 GB (largest of 10 forked workers)

Single-process runs print Peak memory: 401.02 MB.

Why the old number was fiction

It was array_sum() of every worker's peak, plus a snapshot of the main process's current usage taken the moment the first worker exited. Workers never peak at the same instant, so the sum described a moment that never happened. On top of that, two concrete defects:

The main process was understated. We recorded current usage, sampled early, while the main process's real peak comes much later — when it aggregates worker results and writes the result cache. On PHPStan's own self-analysis we recorded 136 MB for a process that actually peaked at 405 MB.

memory_get_peak_usage() is inherited across pcntl_fork(). A child reports its parent's peak before allocating anything:

$a = str_repeat('x', 800 * 1024 * 1024);
unset($a);                            // parent: peak 802 MB, current 2 MB
if (pcntl_fork() === 0) {
    echo memory_get_peak_usage(true); // child that allocated nothing: 802 MB
}

Summed across workers, that multiplied the main process's pre-fork spike — on an incremental run, the loaded result cache — by the number of workers.

What the OS saw, before this change

Two real projects, PHP 8.5, 10 workers. "Largest process" is the kernel's maximum resident set size; the sum column is RSS polled across the whole process tree.

Project / mode printed largest process sum of per-process peaks
Dolor, fork, warm result cache 29.21 GB 3.14 GB 16.69 GB
Dolor, fork, cold cache 12.31 GB 3.04 GB 15.03 GB
Dolor, spawn, cold cache 11.46 GB 2.89 GB 13.98 GB
Lorem, fork, cold cache 7.81 GB 0.98 GB 8.03 GB
Lorem, spawn, cold cache 7.55 GB 0.98 GB 8.13 GB

It was wrong with spawning too. Lorem spawned: 7.55 GB printed against a largest process of 0.98 GB — 7.7×. Dolor spawned: 11.46 GB printed, 2.89 GB largest, and 18% below the sum it was nominally computing (because the main process's contribution was that early snapshot).

Forking made it far worse. Same project, same work — the only difference is that the main process had the result cache loaded when it forked: 29.21 GB printed against a 3.14 GB largest process, 9.3×, and 1.75× even the sum of every process's peak. Nothing on that machine ever held 29 GB.

After this change

Project / mode printed: main largest process (OS) printed: top worker 2nd-largest process (OS)
Dolor, fork, warm 3.07 GB 3.05 GB 2.19 GB 1.48 GB
Dolor, fork, cold 2.92 GB 3.07 GB 1.18 GB 1.34 GB
Dolor, spawn, cold 2.89 GB 2.92 GB 1.13 GB 1.17 GB
Lorem, fork, cold 682 MB 1.13 GB (a worker) 1006.5 MB 1.12 GB
Lorem, spawn, cold 682 MB 0.97 GB (a worker) 968.6 MB 0.96 GB

Where the main process is the largest one, the printed figure matches the kernel's within 1–5%. Worker figures land within 3–13% in spawn mode. Both are per-process, which is also how memory_limit applies — so if a run dies at its limit, these are the numbers that explain why.

What these numbers are not

They are not the memory the run costs the machine. Sampling machine-wide physical memory during these same runs gives ~5.6 GB (Dolor fork, cold) and ~10.3 GB (Dolor fork, warm) — more than any single process, far less than the sum, because forked workers share most of their pages with the parent.

No number derived from memory_get_* can express that: when a forked worker writes to an inherited page, copy-on-write allocates real memory while ZendMM sees nothing. The one honest figure for fork mode is the worker heap number above, which includes memory the worker inherited and still holds — an upper bound on its private footprint (2.19 GB printed against 1.48 GB resident in the warm-cache row).

A machine-level footprint would need OS counters — Pss from /proc/<pid>/smaps_rollup on Linux, phys_footprint on macOS — and is deliberately left for separate work.

The change

  • ForkedProcess calls memory_reset_peak_usage() in the child right after the fork, so a worker reports its own high-water mark instead of its parent's.
  • ParallelAnalyser stops folding the main process into the total; peakMemoryUsageBytes now means "the heaviest worker's peak", and single-process paths report 0.
  • InceptionResult reads this process's peak at the very end — after the result cache is written — and prints both figures with the worker count and mechanism.

The printed number was the sum of every worker's peak plus a snapshot of
the main process's current usage, which described a moment that never
happens: workers do not peak at the same time, and the main process's own
peak comes later still, while it collects the workers' results and saves
the result cache. The snapshot understated it threefold (136 MB recorded
against a 405 MB peak on a self-analysis run).

Fork made it worse: memory_get_peak_usage() carries over into a
pcntl_fork()-ed child, so every worker reported the main process's peak
before allocating anything of its own. Summed across workers that
multiplied a spike the main process took before forking - loading the
result cache on an incremental run - by the number of workers, and one
project displayed 29 GB while no process ever exceeded 3.2 GB.

Report what each process actually reached instead: this process's peak,
read at the very end, and the heaviest worker's - which is also the
number memory_limit applies to, since it is a per-process limit. The
forked child restarts its peak tracking so the figure is its own.

    Peak memory: 3.24 GB (main process), 1.41 GB (largest of 10 forked workers)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014fMtimQECyyBNKTmFddr8L
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant