Bench History Explained

The key: what the run measured

Key idea. A record is keyed by a hash of the paths that can change a measurement, so two commits that differ only in docs share one record, and so do a PR's head and its squashed merge when the branch is up to date with main.

To compare two runs you must know they measured the same engine. A commit hash says too little and too much: a README edit makes a new commit, and a squash merge gives the code a commit its PR never had.

So a record's key names the source, not the commit. MEASURED_PATHS lists the five top-level paths that can change a measurement: src, assets, Cargo.toml, Cargo.lock and .cargo. The source hash is the git tree built from just those entries:

// src/bench/provenance.rs, source_hash: the five paths' tree entries at a revision, as one tree.
let listing = git(repository, &[&["ls-tree", "--full-tree", revision, "--"][..], &MEASURED_PATHS[..]].concat(), None)?;
git(repository, &["mktree"], Some(&entries))

• Every path is classified. A test fails when a new top-level path is in neither MEASURED_PATHS nor the twelve UNMEASURED_PATHS, so a new directory cannot slip out of the hash unnoticed.
• A docs-only commit inherits. 91fda48b and 3f7fd6b5, which changed only README.md, both hash to 2d5d1c36. The M4 Pro's sweep, run on the second, logged "nothing to bench: main's source has a record on every Series here".

The full key adds where and how the run measured:

A record's path is its key, less the Harness version: records/apple-m4-pro-metal/town/2d5d1c36….json. The Harness version is defined under src/, so a source hash already names one. When a second record lands on a path that exists, the quieter of the two stays (chapter 3).

The ruler: Harness version

Key idea. When a change alters how a metric is measured, its Harness version rises, and charts break their line there: a jump across a break is the ruler changing, not the engine.

A Harness version names what the bench's metrics mean. It is 10 on main. Before the history existed nobody numbered it, so praxis history import numbers the commits that changed a metric's meaning, and gives each imported Baseline the version in force at its commit:

From #427's list of Harness version changes, metric names in words.

What does not raise it matters as much. A renamed field that measures the same span, a counter added beside the old ones, and a new pass inside the frame are not breaks: the last is engine work, and the line should show it. Version 8 is the case to remember: the bench started spin-waiting, and town's M4 Pro CPU line fell by about 30% with no engine change at all (Waiting on the GPU Explained).

A scene's content hash breaks a line the same way: a Reference scene given more Instances measures a different load. On the bench page both breaks shade bands or draw a dashed rule, and no change is measured across them.

Quiet records and the Series

Key idea. Every run is kept, but only clean, release-profile runs of main's source on a quiet machine form a Series; the rest are tagged with why they are not on one.

A Series is what a chart draws and what a Regression is found in. A record joins one only if nothing tags it:

• Clean and release. No uncommitted change, and the release profile.
• Main's source. The source hash is main's, or the commit is on main.
• The Series' configuration. 2560×1440 over 1,800 frames; lavapipe's is CI's 640×360 over 64.
• A quiet machine. At the run's start preflight warned of no GPU load over 10%, no other praxis process, no battery power and no busy display, and the run's GPU frame p95 over its median stayed under 1.06.

The last rule is the one a night tests. On the evening of 2026-10-31 the RTX 2070 SUPER's second sweep read p95/median from 1.08 to 1.83 on six of eleven scenes, every one started at 0% GPU load; none of those six joined a Series. Two runs of one key keep the quieter, the lower p95/median, so a noisy evening cannot replace a quiet night's record. lavapipe is the exception: its times mean nothing, so the first record under its key stays, quiet or not.

On 2026-11-02 at 04:00 the M4 Pro's sweep benched main's new source and put all 11 records on their Series. Each Series still begins with the imported Baselines, which is why most of its points are squares on the bench page:

Suspects: a band one bad run cannot widen

Key idea. A record is a Suspect when its frame median sits outside the median of up to 8 records before it, ± the larger of 4 median absolute deviations and 3%; only a step's first record is marked.

A fixed percentage alone, say anything 5% off the last record, fires on noisy passes and misses steady ones, so ticket #435 ruled it out. The band has to follow each Series' own noise.

The textbook band is the mean ± a few standard deviations, but one outlier pulls the mean and inflates the deviation, widening the band it should fall outside. The median and the median absolute deviation (MAD) do not move that way: half the window must change before the median does. Leys and colleagues' 2013 note puts it in its title: "Do not use standard deviation around the mean, use absolute deviation around the median".

center     = median(window)
deviation  = median(|value − center| for each value in window)
tolerance  = max(4 × deviation, 0.03 × center)
suspect    = |median − center| > tolerance

• The window. Up to 8 medians before the record on its line. A line starts at a Harness version, a new scene content or an imported Baseline, and a record with fewer than 3 before it on its line is not judged.
• 4 MADs. For normally distributed noise the standard deviation is about 1.4826 MADs, so 4 MADs is about 2.7 standard deviations.
• The 3% floor. Quiet M4 Pro medians can agree to a few tenths of a percent, which makes the MAD tiny; the floor keeps a 1% wobble from reading as a step.
• The first of a step. After a real step, the window's median lags until five of its eight are at the new level, so the next four records also sit outside. A record continues a step flagged among the 8 before it while it stays on the same side, within the flagged record's tolerance of that record's median, even after a noisy record between them fell back inside the band; a second step on the same side is flagged again.
• Which metrics. The CPU and GPU frame medians, on every Series but lavapipe's, whose times measure no GPU. Faster records are marked too; only slower ones are Regressions.

Each publish rebuilds every Series, so each row carries its marks: "suspects": {"gpu_frame_microseconds": 0.052}, the shift as a share of the window's median. The bench page rings them, red when slower and green when faster.

On the real branch

Rebuilt with PR #487's binary, no live Series has a Suspect: each M4 Pro Series has one or two records benched since its last imported Baseline, and a record needs three before it. Run over the M4 Pro's Baseline history with Baselines not starting lines, a copy of the rule flagged up to 6 records per Series and metric, none on some, each at a commit that re-recorded that Baseline, from 3.5% to 574%. That is the rule finding the steps the Baselines were re-recorded for. It also flagged two CPU frames from the 2026-11-02 night: town's at 14.2% over the median of the three records before it, and crowd's at 15.9% faster.

One detail of the arithmetic: the median is the bench's own nearest-rank percentile, so for an even window it is the lower of the two middle values. That sets the band a little low, so slowdowns are flagged slightly more readily than speedups.

What a Suspect is not yet

Key idea. A Suspect is a candidate, not a Regression: the sweep must re-bench it beside its predecessor on one night, bisect the commits between them, and only then file an issue.

Machine state moves medians between nights, as the RTX's evening showed, so a Suspect compared with records from other nights proves little. #485 has the sweep re-bench the Suspect's source and its predecessor's interleaved on the same night. If the shift holds, it bisects the first-parent commits between them with other source hashes until one is pinned, and a pinned commit starts a new line. #486 then files one needs-triage issue per confirmed Regression, naming the commit, its PR, the Series, the delta and the chart.

MongoDB's performance team made the same split. They inspected results by hand, then added a threshold-based system, and "neither system was sufficient for finding changes in performance in a timely manner"; their system "built upon the E-Divisive means algorithm" now finds change points for a person to triage (Daly and colleagues, ICPE 2020).

What the detector cannot see

• Baselines re-recorded since the history began. Only imported Baselines start a line. A Baseline a PR re-records now does not, so its step will be a Suspect until #485 pins it to that commit and marks it covered.
• A slow drift. A metric that grows 1% a record never leaves a band that follows it. A window of medians finds steps, not slopes.
• A step right after an outlier on the same side. If the outlier sits within the step's tolerance of it, the step reads as continuing the outlier, and the outlier takes the mark. Once #485 clears the outlier, the step should be judged again.

Check yourself

Reading list

Grouped by source, each with what it is good for here. Code links are pinned to e3a96ae0.

Glossary

Bench record, Series, Harness version, Suspect, Reference scene and Baseline are CONTEXT.md's definitions, Bench record shortened; the rest are this guide's own.