[psiha]

Closes #25

What

The method says each result records peak memory, CPU time and thread count, but the runner recorded none of them. This records them, per test, for the runner and the processes it started, so a system that runs in the runner (RAMA, DuckDB) and a server it starts (PostgreSQL, ClickHouse) are measured the same way. New runner/proc.mjs reads /proc. No db.mjs changes.

Per test, an additive resources:

• peakRssUpperBoundMB and peakSource: an upper bound, not a peak at one moment. Each process's VmHWM, reset at the start of the test through clear_refs, added up, although the processes peaked at different times. If the reset is not allowed, the largest resident memory at a scan, and peakSource says so.
• rssSplitMB: anon, file and shmem (RssAnon, RssFile, RssShmem) at the scan where the resident memory was largest. A mapped file counts as file, so compare systems on anon.
• rssMB: afterSetup, afterFirstRun, afterLastRun
• cpuMs (user, sys), involuntaryCtxSwitches, threads
• io: read_bytes and write_bytes of /proc/<pid>/io, with its source named. Reads the page cache served are not counted, and writeBytes is bytes dirtied, not bytes that reached the device.
• diskMB: apparent and allocated size of the entry's folder in --work after the test

And a top-level env: kernel, CPU governor, load average at the start and end, steal time, the type of file system of --work, and the scheduling class of the runner (so a run that was not SCHED_FIFO shows it).

The snapshot's hash now also covers the system version and the RAMA binary named by RAMA_NODE_ADDON_BINARY (its path, size and modification time, not its contents: a rebuild changes the time, and hashing hundreds of MB at every start gains nothing), so a snapshot another build made is rebuilt instead of reused. This makes existing work folders rebuild their snapshot once.

tests/method.md gets the sentence that was not true replaced by what is recorded. runner/README.md explains each field and what it does not cover.

Noise and blind spots it removes

Wall time alone cannot tell a busy machine from a real change. involuntaryCtxSwitches, load, steal time and the scheduling class now say it, and memory, disk and IO are in the results at all.

Schema

Additive: tests.<id>.resources and env. The website, the scores and the expected answers read neither. Off Linux, or with FPABENCH_RESOURCES=0, they are left out (env.source is unavailable without /proc).

Limits, written in the README too

• Shared pages count once for each process that maps them, so a multi-process server such as PostgreSQL is overstated.
• The runner scans /proc before a test, after its setup, after the first and the last run and at its end. It does not scan between the runs in the middle, so no scan lands between two timed reads (a scan takes 0.1 to 0.6 ms). The peak in between is held by each process's own VmHWM.
• A process that starts and ends between two scans adds nothing. One that ends is counted up to its last scan, so a server stopped in teardown loses only the teardown. CPU time of children the runner waited for is not added.
• diskMB counts a reflinked copy of the snapshot in full, so on such a file system it is larger than the space used.
• cpuMs and io cover the whole test (setup, runs, teardown), not the timed part.

How it was checked

On a Linux box, 10m, --quick:

• RAMA (in the runner), again after the changes to the scans: all 29 tests ok and every test has resources. A test that starts from the snapshot shows memory after setup, after the first run and after the last.
• ClickHouse (a server process): L1, W3, R1, R8 ok. The peak (about 1.6 GB, 240 to 390 threads) is far above the runner's own, so the server is counted. R8's afterSetup is the stopped system, far below its afterFirstRun.
• Overhead: not measured on a quiet machine. The box used was busy, so an on or off comparison of FPABENCH_RESOURCES would say nothing. What is checked instead is that the per-run hook reads /proc only after the first and the last run (a unit test). The scans are outside the timer.
• node --test runner/proc.test.mjs checks the parsers on fixed /proc text, including a process name with spaces and brackets, a mount point with an escaped space, and the first-and-last-run rule.
• On Windows the module loads, available is false, and env.source is unavailable. A full run there was not made.

Not checked: PostgreSQL, 100m and 1b (memory sampling cost with a very large process, the size of diskMB walks), macOS, a quiet machine.

🤖 Generated with Claude Code

[Copilot]

Copilot was unable to review this pull request due to rate limiting. Please try again later. See Usage limits for GitHub Copilot for details.