What a frame's GPU time counts

What a frame's GPU time is

Key idea. A frame's GPU time is a span, not a sum: Metal can run passes side by side, so Praxis times each submission of the frame's passes from their earliest start to their latest end, and adds the dynamic TLAS's build.

The GPU has a clock, and each timed pass asks it for the time twice: when the pass starts and when it ends. A pass's time is the difference. A frame's time is not the sum of its passes, because the GPU does not always run them one after another.

Praxis keeps two timestamp queries for each of twelve timed passes, 24 per frame. Most passes write one at their start and one at their end; the dynamic TLAS build is timed between two small marker passes around it, and the BLAS refit from the end of the bend pass to a marker after it. The timestamps land in a ring of buffers: a frame's timestamps are resolved once its submission has completed, FRAMES_IN_FLIGHT + 1 frames later, and the CPU reads them when the ring comes back to their slot, six frames after the frame. In TIMED_PASSES' order:

The span, not the sum

Metal can run a pass alongside the passes recorded around it when neither writes a resource the other binds: its hazard tracking makes a pass that binds a resource wait for earlier writes to it, and one that writes it wait for earlier reads as well (GPU Ordering Explained). The Froxel pass, which lists each cell's lights, and culling write nothing the other binds, so the Froxel pass can start before culling phase one. Summing the passes would count that overlap twice. So frame_times takes, for each submission, the earliest start and the latest end of the passes in it, and calls the difference the span. In the code's words, which give only the first half of Metal's rule:

// From the earliest start to the latest end of `passes` that write both
// timestamps: Metal can run a pass alongside those recorded around it
// when it reads none of their output, so the Froxel pass can start
// before culling phase one, and a pass can end after the last.

The GPU frame total is then the build of the dynamic TLAS, in a submission of its own, plus the span of each submission of the frame's passes. Neither the waits between submissions nor what a submission does before its first pass counts. Chapter 5 is about where the build's time starts.

Frames in flight

The CPU does not wait for a frame to finish on the GPU before recording the next. FRAMES_IN_FLIGHT is 2: "Frames submitted but not finished on the GPU when the next one starts, at most". While the GPU runs frame N, the CPU records frame N+1 and submits it, and the GPU may start N+1's first submission before N's last pass ends. That keeps the GPU busy. It also means a stretch of GPU time can belong to two frames at once, and an accounting has to give it to one of them.

ADR 0019's gate

Key idea. ADR 0019's gate times only what the main pass's design decides, from culling phase one to the end of the lit pass with shadow rays off, and town's Baseline reads 9.8 ms against its 6 ms, so options B and D are prototyped on town before anything changes.

ADR 0019 chose a forward main pass with 4x MSAA, and wrote down when to doubt that choice: when the main pass takes more than 6 ms of GPU on a Reference scene at 1440p. On town the run kept as its Baseline reads 9.8 ms.

The clause, from the ADR's Consequences: "on a Reference scene at 1440p with 4x MSAA, the opaque main pass takes more than 6 ms of GPU, half of ADR 0013's 12 ms, as the median over the scene's camera path, timed from the first culling dispatch to the end of the lit pass with shadow rays off, since every option traces the same rays; then B and D are prototyped on that scene before anything changes." Each part of it is a choice:

• 6 ms. Half of ADR 0013's 12 ms GPU frame Budget. The ADR calls it provisional, as ADR 0013's Budgets are. bench/budgets.json holds it as "gate_microseconds": 6000.
• From culling to the lit pass. Everything the main pass's shape decides: culling, the Froxel pass, the depth pre-pass, the depth pyramid, phase two and the lit pass. Tonemapping and the dynamic TLAS build are outside it.
• Shadow rays off. Options A, B and D all trace the same shadow rays, so the rays cannot tell them apart. With them in, the gate would measure the rays as much as the main pass.
• The median over the camera path. One number per scene, from all 1,800 measured frames; the p95 is reported beside it.

In code

frame_times starts the gate at culling phase one's start timestamp and ends it at the latest end of the passes from culling phase one to the lit pass. It starts at culling, not at the earliest of those passes, even though the Froxel pass can start first on Metal:

let culling_phase_one = pass_position(CullingPhase::One.label());
let (_, gate_end) = bounds(culling_phase_one..pass_position(LIT_PASS) + 1);
// ...
gate_time: time(ticks[2 * culling_phase_one], gate_end),

praxis bench now renders each scene twice, once with shadow rays on and once with RenderSwitches { shadow_rays: false }, and the gate's median and p95 come from the second run. praxis compare fails a gate past 6 ms, and one grown more than 10% and 20 µs past its Baseline; past 5% it warns. praxis report, which prints a bench run as markdown tables against its Baseline and the Budgets, gives the gate a row of its own, with the 6 ms in its Budget column. texture-wall's Baseline reads 14.3 ms, so compare fails its gate too, though its timed passes cover little of that span; M3's exit criterion 2 judges the gate on town.

town breaches it

The records give the gate two medians, from two runs. The Baseline's 9,821 µs, which praxis report prints as 9,820.7, comes from one of five runs recorded after the gate's start moved to culling phase one: the one kept for its lowest CPU frame. The owner's decision, posted on #207 and on map #17, quotes 9.61 ms (p95 10.15) from a run made before that move, when the gate still began at the earliest start among its passes: the Froxel pass's, in frames where Metal ran it first. Both lie within the seven runs' 9.6 to 10.3 ms.

town's frames draw a median of 137,730 Instances and 5,552,214 triangles, with 10,000 Moving Instances. The two extra runs, measured the same way as the decision's, rule two suspects out. Emptying the dynamic TLAS removes its build, which runs beside the frame (chapter 5): the gate reads 9.58 ms against that run's 9.61. One frame in flight stops frame N+1's work from overlapping frame N's: 9.58 ms again. The breach is the main pass's own work.

The lit pass alone would have to lose three quarters of its 5.15 ms for the Baseline's gate to pass with nothing else changed. That is what the clause's second half is for: before anything changes, the two other shapes of main pass are built on town and measured.

Options B and D

Key idea. B adds a per-sample ID so the lit pass draws only the triangles that own a sample, D shades from those IDs in compute and must work out texture derivatives itself, and each prototype answers with its gate on town and its image beside A's.

Today's main pass, option A, rasterizes each surviving triangle twice: once for depth, once to shade at depth equal. B adds a record of which triangle won each sample. D goes further and shades from that record in compute, with no second raster.

Both start from a visibility pre-pass: depth plus a 32-bit Instance-and-triangle ID per sample, in an R32Uint target. At 4x MSAA and 2560×1440 that is 58,982,400 bytes, the research doc's computation. On the M4 Pro, writing the triangle's ID through primitive_index made the ID pass 19 to 24% slower than one writing the Instance alone.

B is A plus one attachment, and keeps A's image quality: the lit pass rasterizes the same surviving triangles at depth equal. Its bet is that the scan's compaction saves the lit pass more than the IDs cost. D removes the second raster and the quad overdraw that comes with shading in 2×2 quads, but takes on derivatives by hand, since compute has no quads to take differences across. A visibility buffer shaded in quads anyway, as Nanite's material pass was in 2021, does not escape this: its quads span depth discontinuities, UV seams and different objects, and finite differences across those are, in Karis's word, "nonsense", often picking far too coarse a mip level.

The owner's decision follows the clause: #296 prototypes B and #297 prototypes D, both on town, and both block M3's exit (#211). Each prototype's job is the gate's number for its shape, rays off, on the same camera path, and the image beside A's.

Shadow-ray time

Key idea. Shadow rays change several passes, so their time is each frame's GPU frame total with rays on minus the same frame's with them off, and because that difference can read below zero, growth from a Baseline at or below zero is judged by how much it grew, not by a ratio.

The gate leaves the rays out, so ADR 0019 reports them beside it as a line of their own. Praxis has no pass that is only "the rays": they change several passes. So their time is the frame with rays minus the same frame without.

Both of bench's runs render the same Reference scene from the same camera path with the same injected time, so frame 700 of one run is frame 700 of the other. For each measured frame, shadow_ray_microseconds takes the GPU frame total with rays on less the one with rays off, and reports the median and p95 of those differences.

On town the lit pass reads 8.7 ms with rays on against 5.15 with them off, and 8.66 ms with one frame in flight, so its extra 3.5 ms is not overlap with the frames on either side. Nor can it be read as the rays' cost inside the lit pass: it is more than the 3.2 ms the whole frame gains, and the shadow mask pass reads 4.6 ms besides, so with rays on these passes' readings overlap within the frame. On the other scenes, reading the shadow mask cost the lit pass 22 to 49 µs (#260). Where town's ray time goes is not yet measured.

Why it can read below zero

Each GPU frame total carries its own run-to-run noise. Where the rays cost almost nothing, the difference of two noisy numbers is centred near zero, and half the frames can read negative; the median can too. A negative time is no bug here, but it breaks the usual test for a regression, which divides the current median by the Baseline's: a ratio against zero or a negative number means nothing.

So check_regression passes anything at or below its Baseline, and from a Baseline at or below zero treats any growth as past every ratio. It still fails only when the growth also exceeds the 20 µs noise floor for GPU times:

if current <= baseline {
    return;
}
// From a Baseline at or below zero, any growth is past every ratio.
let ratio = if baseline > 0.0 {
    current / baseline
} else {
    f64::INFINITY
};
let verdict = if ratio > FAIL_RATIO && current - baseline > noise_floor {

Its test reads the rule back: from a Baseline of −10 µs, −40 passes and 600 fails. praxis report prints shadow-ray time in a row of its own, with no Budget, and shows a dash in its Ratio column over a Baseline at or below zero, where a ratio means nothing.

Work hiding behind the frame before

Key idea. Work that runs beside the frame before should be counted once, so the dynamic TLAS build now counts only from the end of that frame's last pass: on town it reads 48 µs, against 4.1 ms when the overlap was counted twice, and takes 665 µs run alone.

The dynamic TLAS build runs in a submission before frame N+1's passes. With two frames in flight, Metal starts it while frame N's lit and tonemapping passes are still running, since those read no TLAS. Where the build's time starts decides whether that overlap is counted once or twice.

The build's start timestamp is the end of a small marker pass that queries the TLAS just before it (Dynamic TLAS Explained, chapter 7, shows why a marker must query the TLAS), and it often falls deep inside frame N. Counting from there would charge frame N+1 for time frame N's passes also spent. So frame_times takes a floor: the build counts from the later of that timestamp and the floor. The question is which floor.

• Before: the end of frame N's shadow mask pass. The last pass that reads a TLAS. On town the build then read 4.1 ms, and the GPU frame total, build plus span, read 17.6 to 18.1 ms: the stretch where the build ran beside frame N's lit and tonemapping passes was in frame N's span and again in frame N+1's build.
• Now: the end of frame N's last pass (last_pass_end), the owner's decision. The build counts only what sticks out past frame N. On town it reads 48 µs, and the total 13.6 ms (13,568 µs median, p95 14,188).

The readback keeps each frame's last pass end, and hands it to the next frame only if that frame directly follows; otherwise the floor is zero:

// The dynamic TLAS's build runs alongside the frame
// before it, which hides most of it, so it counts
// from the end of that frame's last pass.
let previous_pass_end = self
    .last_pass_end
    .filter(|&(previous, _)| previous + 1 == frame)
    .map_or(0, |(_, tick)| tick);

Three readings of one build

48 µs is not the build's cost; it is what of the build sticks out past frame N. Run alone it takes 665 µs. What it costs the passes it runs beside is not yet separated: with the dynamic TLAS emptied, the frame total with rays on read 12.8 ms against 13.6, but that 0.8 ms (this page's subtraction) also holds what the shadow rays spend on the 10,000 Moving Instances, which an empty dynamic TLAS no longer holds. Whatever the build takes from those passes now shows up in their times instead of in the build's.

On the orrery and the crowd the build is hidden entirely and reads 0. Their GPU frames fell from 2,314 to 2,140 µs and from 3,287 to 2,679 µs: no work changed, only what is counted twice.

The ceilings

ADR 0020 caps the dynamic TLAS. The owner set the ceilings from these figures, for M3's exit criterion 6. compare enforces both on every scene, and praxis report prints them in its Budget column, the GPU one on the dynamic_tlas_build row:

The CPU side reaches its ceiling under load: 0.96 ms at a load average of 2.8, and 1.04 to 1.15 ms at 3 to 3.6; the top of that range is past the 1.1 ms ceiling. The CPU figure in the table is the Baseline's. The GPU figure, 48 µs, is the reading the decision and docs/performance.md's text give; the Baseline's own, which praxis report prints in that doc's table, is 48.6 µs (p95 462.8).

Checking frames far along a path

Key idea. To check frames far along a camera path, draw frame 0, which anchors the static TLAS, and each checked frame with the one before it, whose depth culling reads, and run the World through every other frame undrawn.

Culling must never change the image: a frame drawn with culling on must match the same frame with it off. One frame is not enough for town, whose camera walks a street, rises over the rooftops and circles. Drawing every frame up to 1,800 to check eight of them is too slow, so the test draws only what each checked frame depends on.

On CI's lavapipe town takes about 0.4 s a frame; drawing to frame 1,800 for each of four variants would take most of an hour. praxis::capture_frames(context, scene, shift, width, height, frames, switches) draws three kinds of frame and runs the World through the rest:

• Frame 0, always. Its camera anchors the static TLAS. Without it, pbr-sweep's orbit anchored the TLAS in another 64 m cell, and 107 pixels of shadow speckle differed.
• Each listed frame and the frame before it. Culling's phase one draws what was visible the frame before, read from that frame's depth. Drawing the frame before gives phase one the same input as in an unbroken run.
• Every other frame, undrawn. State::simulate runs the frame's Ticks and moves the camera, and prepares nothing. The next prepare writes every change since the last.

One pixel on lavapipe

On lavapipe, one sample of one pixel, (635, 242) at frame 1200, differed between culled and unculled, by 12. On Metal the frames match exactly. A missing Instance, culled when it should be drawn, is the bug this check exists to catch, so the probes (#300) asked whether one was missing:

• Culled variants with different splits between the two phases: identical to each other.
• Occlusion culling off: still differs.
• The frustum widened by about 16 pixels, with occlusion: matches.
• The same widened frustum without occlusion: differs again.

The last probe settles it. A widened frustum without occlusion draws a strict superset of the Instances the matching probe drew. If the difference came from a missing Instance, drawing more could not bring it back. It came back, so lavapipe resolves that sample by what else the draw holds, not by whether the right Instance is in it. The decision: on software Vulkan the culling check allows one pixel per frame (SOFTWARE_UNCULLED_PIXELS = 1); Metal, the reference, must match exactly.

Check yourself

Three questions on the guide's central ideas. Try each before you open its answer.

Why does Praxis time a frame's passes as a span from their earliest start to their latest end, rather than adding up each pass's time?

On Metal a pass can run alongside the passes recorded around it when neither writes a resource the other binds, so the Froxel pass can even start before culling phase one. Adding the passes' times would count that overlap twice. Taking each submission's earliest start and latest end counts that shared stretch once, and the GPU frame total adds the dynamic TLAS's build to those spans.

Why does ADR 0019's gate leave shadow rays out, and how does Praxis still measure what they cost?

Options A, B and D all trace the same shadow rays, so the rays cannot tell them apart; with them in, the gate would measure the rays as much as the main pass. No pass is only the rays, so bench renders each scene twice and takes each frame's GPU frame total with rays on less the same frame's with them off. Where the rays cost almost nothing that difference can read below zero, so growth from a Baseline at or below zero is judged against the 20 µs noise floor, not by a ratio.

On town the dynamic TLAS build once read 4.1 ms of GPU and now reads 48 µs. What changed, and is 48 µs what the build costs?

Only the accounting changed: the build runs beside frame N's lit and tonemapping passes, and counted from the end of frame N's shadow mask pass, that overlap sat in both frame N's span and frame N+1's build, so the GPU frame total read 17.6 to 18.1 ms instead of 13.6. Counted from the end of frame N's last pass, 48 µs is only what of the build sticks out past frame N, not its cost. Run alone, with one frame in flight, the build takes 665 µs, and what it takes from the passes beside it now shows in their times, not yet separated.

Reading list

Grouped by source, each with what it is good for here. Repo links are pinned to 800c0d2, the commit that merged PR #299.

Glossary

Where CONTEXT.md defines a term, the definition here and in the tooltips is its own at 800c0d2, or a quoted part of it; the GPU frame total and ADR 0019's gate quote its Frame stats entry, the gate adding what ADR 0019 says of it. TLAS, dynamic TLAS and submission follow the earlier volumes, and timestamp query extends theirs to render passes; frames in flight quotes the code.