Skip to content

feat: tools to compare two builds fairly, with their spread #48

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
psiha wants to merge 1 commit into
base: main
Choose a base branch
from
Open

feat: tools to compare two builds fairly, with their spread #48

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
45 changes: 45 additions & 0 deletions tools/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
# Tools

## Comparing two builds

The runner times one system once. A difference between two runs of two builds can be the builds, or the order they ran in, a neighbour on the machine, a different disk, or a snapshot one build made for the other. `ab.sh` takes the last four out, and `summarize-ab.mjs` says how large the difference is next to the noise between runs of the same build. Linux, bash 4.4 or later.

```bash
tools/ab.sh --entry rama --dataset ~/data/fpabench-10m --work ~/ab/work --out ~/ab/out \
--arm old=$PWD,RAMA_PATH=$HOME/rama-old,RAMA_NODE_ADDON_BINARY=$HOME/rama-old/rama.node \
--arm new=$PWD,RAMA_PATH=$HOME/rama-new,RAMA_NODE_ADDON_BINARY=$HOME/rama-new/rama.node \
--legs 3 --tests R1,W3 -- --expected ~/run/expected.json
node tools/summarize-ab.mjs ~/ab/out
```

An `--arm` is `NAME=<FPABench checkout>`. Each `--env NAME=VALUE` after it is an environment variable for that arm; a value may contain commas. Everything after `--` goes to `runner/run.mjs` as it is. The other options:

| Option | What it does |
| --- | --- |
| `--entry`, `--dataset`, `--tests`, `--machine` | As in the runner. `--dataset` is a folder or a published size. |
| `--work` | Each arm gets its own folder in it. Downloaded datasets are shared. |
| `--out` | One folder per leg, `<arm>-L<n>`, with the runner's files, `run.log`, and `leg.txt`: the host, the load, the scheduling and the exit code. |
| `--legs` | How many times each arm runs. The default is 6, the least is 5. |
| `--timeout` | Seconds a leg may take. A leg over it is a failed leg, never a wait. |
| `--cpus` | Pin the runs to these CPUs, such as `4-23`. At least 2. Without it the runner and any server it starts inherit `SCHED_FIFO` 50 on every CPU, and can starve the rest of the machine: pass it on a machine that has other work. |

What it does:

- **Own work folder.** Every arm has its own `--work` folder, and one untimed load of its own first, so no arm times a snapshot another build made.
- **Turns.** The legs run `A B`, `B A`, `A B`, `B A`, ... (the runs go A B B A A B B A), so neither arm is always first or last. With three or more arms each leg is the next row of a rotating Latin square. The order is fully balanced when `--legs` is a multiple of the number of arms, and `ab.sh` warns when it is not. Running again into the same `--out` replaces the legs of the arms.
- **Real-time.** Each leg tries `chrt -f 50`, then the same through `sudo -n`. The check runs the same command as a leg (the prefix, the pin, `env` with a variable and `PATH`, then `node`) and trusts it only when the process finds `node`, sees the variable, and reports class `FF` and a priority above 0 in `ps`. The arm's variables are set after `sudo`, which would otherwise drop them. Otherwise the leg runs normally and is marked `not realtime` in `leg.txt` and in the summary.
- **Same machine.** It records the host name, the boot id and the machine id, and stops when any of them changes between legs.

The summary prints, per test and arm, the median over the legs and its spread, which is (max - min) / median. Two arms are "beyond spread" only when there are at least 5 legs in each and all three hold:

1. the ranges (min to max) of their legs do not overlap;
2. the difference of the medians is larger than the range (max - min) of each of the two arms;
3. that difference is at least 1% of the first arm's median.

Anything else is "within spread", "fewer than 5 legs", or "check failed in a leg".

**False alarms.** If both arms were the same build, rule 1 alone flags a test with probability 2 / C(2n, n) for n legs each: 10% at 3 legs, 0.8% at 5, 0.2% at 6. Rules 2 and 3 only lower that; simulated with normally distributed noise, the three rules together flag 0.2% of the tests at 5 legs and 0.04% at 6. That is per test: over m tests the chance of at least one false flag is 1 - (1 - p)^m, so for 20 tests at 5 legs about 15% by rule 1 alone and about 3.5% with all three rules. It says nothing about a difference smaller than the spread: that is "within spread", not "no difference".

It refuses results from machines that differ in their CPU, CPU count, memory, kernel, Node version, dataset or tests version. `--csv` prints the same numbers for a spreadsheet.

A timed run needs a machine nothing else uses. `ab.sh` does not check that, and a run with other work on the machine shows up as a large spread.
166 changes: 166 additions & 0 deletions tools/ab.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,166 @@
#!/bin/bash
# Runs two or more builds of the same entry against each other, in turns, on this machine.
#
# tools/ab.sh --entry rama --dataset ~/data/fpabench-10m --out ~/ab/out --work ~/ab/work \
# --arm A=/path/to/checkout --env RAMA_PATH=/path/a --env RAMA_NODE_ADDON_BINARY=/path/a/rama.node \
# --arm B=/path/to/checkout --env RAMA_PATH=/path/b --env RAMA_NODE_ADDON_BINARY=/path/b/rama.node \
# [--legs 6] [--tests R1,W3] [--machine <name>] [--timeout <seconds>] [--cpus 4-23] \
# -- --expected ~/run/expected.json
#
# An arm is `--arm NAME=<FPABench checkout>`, followed by any number of `--env NAME=VALUE` for that arm (a
# value may contain commas). The checkout may be the same for every arm (two builds of one system) or
# differ (two versions of the runner). What follows `--` goes to runner/run.mjs as it is, such as
# --expected and --quick.
#
# Each arm has its own work folder. Each gets one untimed load first, so no arm times a snapshot another
# one made. Then the legs run in turns that keep the order balanced (A B, B A, A B, ...; for 3 or more arms
# a rotating Latin square), so that no arm is always first or last. At least 5 legs per arm.
# Each leg runs under SCHED_FIFO 50 when the machine allows it, and the result says whether it did. Without
# --cpus the runner and any server it starts get FIFO 50 on every CPU and can starve the rest of the
# machine: pass --cpus on a machine that has other work. The run stops when the machine changes between legs.
# Never time on a machine that other work uses. Running again into the same --out replaces the legs of the arms.
# The results are in <out>/<arm>-L<leg>/, and tools/summarize-ab.mjs prints the table.

set -euo pipefail

here=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
entry= dataset= out= work= machine= tests= cpus= legs=6 leg_timeout=7200
arms=() arm_envs=()
while [ $# -gt 0 ]; do
case $1 in
--entry) entry=$2; shift 2 ;;
--dataset) dataset=$2; shift 2 ;;
--out) out=$2; shift 2 ;;
--work) work=$2; shift 2 ;;
--arm)
case $2 in ?*=?*) ;; *) echo "--arm needs NAME=<checkout>, not $2" >&2; exit 2 ;; esac
arms+=("$2"); arm_envs+=(""); shift 2 ;;
--env)
[ ${#arms[@]} -gt 0 ] || { echo "--env comes after the --arm it belongs to" >&2; exit 2; }
case $2 in [A-Za-z_]*=*) ;; *) echo "--env needs NAME=VALUE, not $2" >&2; exit 2 ;; esac
arm_envs[${#arms[@]}-1]+="$2"$'\n'; shift 2 ;;
--legs) legs=$2; shift 2 ;;
--tests) tests=$2; shift 2 ;;
--machine) machine=$2; shift 2 ;;
--timeout) leg_timeout=$2; shift 2 ;;
--cpus) cpus=$2; shift 2 ;;
--) shift; break ;;
*) echo "unknown option $1" >&2; exit 2 ;;
esac
done
extra=("$@")
if [ -z "$entry" ] || [ -z "$dataset" ] || [ -z "$out" ] || [ -z "$work" ] || [ ${#arms[@]} -lt 2 ]; then
sed -n '2,10p' "$0" | sed 's/^# \{0,1\}//' >&2
echo "needs --entry, --dataset, --out, --work and at least two --arm" >&2
exit 2
fi
[[ $legs =~ ^[0-9]+$ ]] && [ "$legs" -ge 5 ] || { echo "--legs $legs: at least 5 legs per arm (fewer cannot tell a difference from noise)" >&2; exit 2; }
if [ $((legs % ${#arms[@]})) != 0 ]; then echo "WARNING: $legs legs is not a multiple of ${#arms[@]} arms: the order of the arms is not fully balanced" >&2; fi
case $dataset in 10m|100m|1b|10b) ;; *) dataset=$(cd "$dataset" && pwd) ;; esac
mkdir -p "$out" "$work/datasets"
out=$(cd "$out" && pwd)
work=$(cd "$work" && pwd)

# --- the machine: the legs must all run on the one ---------------------------------------------------------

fingerprint() { echo "$(hostname) $(cat /proc/sys/kernel/random/boot_id 2>/dev/null || echo no-boot-id) $(cat /etc/machine-id 2>/dev/null || echo no-machine-id)"; }
host=$(fingerprint)
echo "$host" > "$out/host.txt"
check_host() { [ "$(fingerprint)" = "$host" ] || { echo "the machine changed ($(fingerprint), it was $host): stopping" >&2; exit 3; }; }

# A single CPU would starve the runner and a server of each other under a real-time class.
if [ -n "$cpus" ]; then
n=0
IFS=, read -ra parts <<< "$cpus"
for p in "${parts[@]}"; do case $p in *-*) n=$((n + ${p#*-} - ${p%-*} + 1)) ;; *) n=$((n + 1)) ;; esac; done
[ $n -ge 2 ] || { echo "--cpus $cpus is one CPU: a real-time runner needs at least 2" >&2; exit 2; }
fi
pin=()
[ -z "$cpus" ] || pin=(taskset -c "$cpus")

# --- real-time scheduling, verified instead of assumed -----------------------------------------------------
# What the shell can start under SCHED_FIFO 50 is tried directly, then through sudo. The probe has the shape
# of a leg: the prefix, the pin, then env with a variable and PATH, then a process that must find node, see
# the variable and report its class. A candidate counts only when that process reports class FF and a
# non-zero real-time priority.

rt=()
sched="not realtime"
probe() { "$@" "${pin[@]}" env AB_PROBE=seen HOME="$HOME" PATH="$PATH" sh -c 'echo "$AB_PROBE $(node --version >/dev/null 2>&1 && echo node) $(ps -o cls=,rtprio= -p $$)"' 2>/dev/null | tr -s ' ' | sed 's/^ //; s/ $//'; }
for candidate in "chrt -f 50" "sudo -n chrt -f 50 setpriv --reuid=$(id -u) --regid=$(id -g) --clear-groups --"; do
read -ra c <<< "$candidate"
got=$(probe "${c[@]}" || true)
if [[ $got =~ ^seen\ node\ (FF\ [1-9][0-9]*)$ ]]; then rt=("${c[@]}"); sched=${BASH_REMATCH[1]}; break; fi
done
[ ${#rt[@]} -gt 0 ] || echo "WARNING: SCHED_FIFO is not available (no rtprio limit, no passwordless sudo, or node not found under it): the legs run \"not realtime\" and say so" >&2
{ [ ${#rt[@]} -eq 0 ] || [ -n "$cpus" ]; } || echo "WARNING: without --cpus the runner and any server it starts run FIFO 50 on all CPUs" >&2
echo "scheduling: $sched"

# --- one leg ------------------------------------------------------------------------------------------------

arm_name() { echo "${1%%=*}"; }
arm_dir() { realpath "${1#*=}"; }

# run_leg <arm index> <folder name> <tests> <real-time 0|1>; the runner gets what followed `--`
run_leg() {
local idx=$1 name=$2 only=$3 realtime=$4
local spec=${arms[$1]} dir envs=() leg=$out/$2 prefix=() tflag=()
dir=$(arm_dir "$spec")
while IFS= read -r kv; do [ -z "$kv" ] || envs+=("$kv"); done <<< "${arm_envs[$idx]}"
[ "$realtime" = 1 ] && [ ${#rt[@]} -gt 0 ] && prefix=("${rt[@]}")
[ -z "$only" ] || tflag=(--tests "$only")
rm -rf "${leg:?}"
mkdir -p "$leg"
{
echo "host=$(fingerprint)"
echo "load=$(cut -d' ' -f1-3 /proc/loadavg 2>/dev/null || echo unknown)"
echo "sched=$([ ${#prefix[@]} -gt 0 ] && echo "$sched" || echo "not realtime")"
echo "start=$(date -u +%FT%TZ)"
echo "run=$((++runs))"
} > "$leg/leg.txt"
local rc=0
# env comes AFTER the real-time prefix: sudo resets the environment, so assignments before it never
# arrive. PATH and HOME are passed on for the same reason; an arm's own PATH or HOME wins.
(cd "$dir" && timeout "$leg_timeout" "${prefix[@]}" "${pin[@]}" env HOME="$HOME" PATH="$PATH" "${envs[@]}" node runner/run.mjs --entry "$entry" --dataset "$dataset" \
--work "$work/$(arm_name "$spec")" --out "$leg" ${machine:+--machine "$machine"} "${tflag[@]}" "${extra[@]}") > "$leg/run.log" 2>&1 || rc=$?
{ echo "end=$(date -u +%FT%TZ)"; echo "rc=$rc"; } >> "$leg/leg.txt"
return $rc
}

# The arms, in order, for leg i (0-based). Two arms: A B, B A, A B, ... so the runs go A B B A A B B A.
# Three or more: row i of a Latin square, the cyclic one over the sequence 0 1 k-1 2 k-2 ... shifted by i,
# which for an even k also balances which arm follows which when legs is a multiple of k.
leg_order() {
local i=$1 k=${#arms[@]} j seq=(0) lo=1 hi=$((${#arms[@]} - 1))
while [ $lo -le $hi ]; do
seq+=($lo); lo=$((lo + 1))
if [ $lo -le $hi ]; then seq+=($hi); hi=$((hi - 1)); fi
done
for j in "${seq[@]}"; do echo $(((j + i) % k)); done
}

failed=0 runs=0
for idx in "${!arms[@]}"; do
name=$(arm_name "${arms[$idx]}")
rm -rf "${out:?}/$name-prep" "${out:?}/$name"-L[0-9]*
mkdir -p "$work/$name"
# A real folder where the link goes (an earlier run, or a runner that made one) is removed when empty,
# and never linked into.
if [ -e "$work/$name/datasets" ] && [ ! -L "$work/$name/datasets" ]; then
rmdir "$work/$name/datasets" || { echo "$work/$name/datasets is a folder with files in it: move it away" >&2; exit 2; }
fi
ln -sfn "$work/datasets" "$work/$name/datasets"
echo "prepare $name: one untimed load"
check_host
run_leg "$idx" "$name-prep" L1 0 || { echo "the load of $name failed, see $out/$name-prep/run.log" >&2; exit 1; }
done
for ((i = 0; i < legs; i++)); do
for idx in $(leg_order $i); do
name=$(arm_name "${arms[$idx]}")
check_host
echo "leg $((i + 1)): $name"
run_leg "$idx" "$name-L$((i + 1))" "$tests" 1 || { echo "leg $((i + 1)) of $name failed (rc $?), see $out/$name-L$((i + 1))/run.log" >&2; failed=1; }
done
done
node "$here/summarize-ab.mjs" "$out" || true
exit $failed
Loading
Loading