[psiha]

What

Two tools to compare two builds of an entry fairly, in a new tools/ folder (nothing else changes):

• tools/ab.sh runs the arms in turns on one machine:
  • every arm has its own work folder and one untimed load of its own, so no build times a snapshot another build made
  • the legs run A B, B A, A B, so neither arm is always first or last
  • each leg tries SCHED_FIFO 50 and trusts it only when ps reports class FF and a non-zero priority for a process it started. If it cannot get it, the leg runs normally and says not realtime, in the output and in leg.txt
  • it stops if the host, boot id or machine id changes between legs, and every leg has a timeout
• tools/summarize-ab.mjs prints, per test and arm, the median over the legs and the spread ((max - min) / median). Two arms are "beyond spread" only when the legs' ranges do not overlap, with at least 3 legs each. It refuses results from different machines, datasets or tests versions. --csv for a spreadsheet.

Usage is in tools/README.md.

Noise it removes

Order bias (the second run profits from a warm cache or suffers from heat), a shared snapshot between builds, scheduler jitter, a machine that changed in between, and having no spread to judge a difference against.

Schema

None. It reads the existing result files and does not touch runner/run.mjs.

How it was checked

On a Linux box, 10m, --quick, the in-process RAMA entry, tests R1 and W3, 3 legs, the same build as both arms:

• scheduling verified as FF 50 in all legs, leg order AB BA AB, exit codes 0, and the summary table printed. The box was busy, so W3 showed a wide spread and, correctly, no difference beyond it.
• with the real-time limit lowered to 0, a warning is printed, the legs are marked not realtime, and the summary shows it. With 2 legs it says "fewer than 3 legs" instead of a verdict.
• the summary on fabricated result folders: a clear difference is "beyond spread", an overlapping one "within spread", and a folder from a different CPU is refused.

Not checked: the stop on a changed machine (the summary's refusal was), a timed run on a quiet machine, the sudo path, --cpus, a run with a failing leg, macOS.

🤖 Generated with Claude Code

[Copilot]

Copilot was unable to review this pull request due to rate limiting. Please try again later. See Usage limits for GitHub Copilot for details.