Waiting on the GPU Explained

Where the CPU waits

Key idea. With two frames in flight and a GPU-bound frame, the main thread waits for the oldest frame in every frame; it can wait by blocking, asleep until the GPU signals, or by spinning, polling on its core.

The CPU records a frame and hands it to the GPU, then starts the next. It may run ahead by a fixed number of frames. When the GPU is the slower of the two, the CPU reaches that limit every frame and has to wait.

Praxis lets two frames be in flight. FRAMES_IN_FLIGHT is 2, in src/render/mod.rs: "Frames submitted but not finished on the GPU when the next one starts, at most: the window's swapchain latency and capture's queue." While the GPU runs frame N, the CPU records frame N+1 beside it (Frame Timing Explained). Before it starts N+2, frame N must be done.

On town at 2560×1440 with 4x MSAA, the GPU frame is now 20.1 ms, since volume 26's sun shadow edge rays (Shadow Edge Rays Explained). Spin-waiting, the CPU frame reads 4,977 µs. So the main thread spends most of each frame waiting.

Two places, two ways

The window and the bench wait in different places. The window waits in the surface's get_current_texture, in src/lib.rs, for a swapchain texture to come free. praxis bench, praxis capture and the Goldens all render through render_frames in src/capture/mod.rs, which keeps a queue of submissions and, when two are in flight, waits for the oldest with device.poll:

// src/capture/mod.rs at 958ed4e9, render_frames.
if in_flight.len() == FRAMES_IN_FLIGHT {
    let oldest = in_flight.pop_front();
    let timeout = match wait {
        FrameWait::Block => None,
        FrameWait::Spin => Some(Duration::ZERO),
    };
    loop {
        match context.device.poll(wgpu::PollType::Wait {
            submission_index: oldest.clone(),
            timeout,
        }) {
            Ok(_) => break,
            Err(wgpu::PollError::Timeout) => std::hint::spin_loop(),
            Err(error) => return Err(error.into()),
        }
    }
}

• Blocking, timeout: None. On Metal, wgpu-hal 30's fence wait is a parking_lot condition variable that the command buffer's completion handler signals. The thread sleeps until the frame is done: no sleep loop, no polling. Its core is free for other work, or to go idle.
• Spinning, timeout: Some(Duration::ZERO). Each poll returns at once with PollError::Timeout until the frame is done, and the loop polls again, with std::hint::spin_loop() between. The thread never leaves its core.

Before PR #400, render_frames always blocked, as the window does. Neither way changes what the GPU does: it gets the same frames in the same order. What the measurements in chapter 4 found is that the choice changes how fast the CPU's next frame runs.

Cores, clusters and clocks

Key idea. This M4 Pro's cores sit in clusters of four that share an L2 cache, and the kernel's scheduler may adjust each cluster's frequency depending on its workload, its thermal pressure and other factors; how it weighs a thread that sleeps for most of each frame is not published.

This M4 Pro has 12 cores of two kinds. sysctl reports them by performance level:

hw.perflevel0.name: Performance
hw.perflevel0.logicalcpu: 8
hw.perflevel0.cpusperl2: 4
hw.perflevel1.name: Efficiency
hw.perflevel1.logicalcpu: 4
hw.perflevel1.cpusperl2: 4

Eight performance cores in two clusters of four, and four efficiency cores in one; each cluster shares one L2. The core numbers below are the ones the measurement patch read (chapter 3).

Who sets the clock

Apple's Tech Talk Tune CPU job scheduling for Apple silicon games says: "Each cluster may be independently activated or have its frequency adjusted by the kernel's scheduler -- depending on the current workload, the current thermal pressure for that cluster, and other factors." A core's speed is therefore not fixed: the same code can run faster or slower depending, in Apple's words, on the workload, the thermal pressure and other factors.

The same talk names two costs of a thread that waits. "[W]hen a CPU core has nothing to do for quite some time, it goes idle to save energy. And reactivating an idle core takes a little bit of time. That is our first cost, the core wake-up cost." And: "When you signal and wait for threads, that generally means some will be scheduled on the CPU core and some will be blocked and moved off core. Doing that many times is a performance pitfall."

What the talk does not say, and what no Apple source the ticket found says, is how the scheduler weighs a thread that blocks for most of each frame and runs for the rest. This guide does not guess. It reports what the chain's timing shows the core's speed did, and leaves the rule that set it unexplained.

QoS and priority

A thread on macOS also carries a quality-of-service (QoS) class and a scheduler priority. Either could, in principle, keep a thread off a slow core or a slow clock. The ticket named the QoS class as a suspect, with frequency scaling and migration to an efficiency core. Chapter 4 tests QoS, priority and migration.

Measuring a clock without counters

Key idea. Without access to the clock, time a fixed piece of work: if the work never changes, a change in its time is a change in the core's speed.

macOS's powermetrics reports each cluster's frequency, but it needs root, which the measurement did not have. So the patch read the clock through work. It timed a fixed chain of 2,000 dependent multiply-adds on the main thread:

// The measurement patch, not committed.
fn work() -> f64 {
    let started = std::time::Instant::now();
    let mut value = 0x1234_5678u64;
    for _ in 0..2_000 {
        value = value.wrapping_mul(6364136223846793005).wrapping_add(1442695040888963407);
    }
    std::hint::black_box(value);
    started.elapsed().as_secs_f64() * 1e6
}

• Dependent. Each step needs the value the step before produced, so the core cannot run steps side by side; black_box keeps the compiler from removing a result nothing reads. The chain does the same work every time it runs.
• Three points a frame. The patch ran it at each frame's start, after the wait; after update; and after render. A clock that changes within a frame shows as a difference between the three.
• About 2.0 µs at full clock. Spin-waiting, the chain's medians read 1.92 to 2.04 µs at all three points.

Which core

A slow chain could mean a slow clock on a performance core or a run on an efficiency core, so the patch also recorded the core. It read the low 12 bits of the arm64 register TPIDR_EL0, which gave 0 to 11 on this machine, and mapped the numbers to clusters by timing the same chain on each core: 0 to 3 were slower, the efficiency cluster; 4 to 7 and 8 to 11 the two performance clusters. Apple documents no such use of the register. Read it as what the patch did on this machine, an observation, not an interface to build on.

The patch also logged each CPU stage's median, and switched by environment variable the frames in flight, a spin-wait (device.poll with a zero timeout, in a loop) and a priority of 45 set with pthread_setschedparam under the default time-sharing policy. Its core counts span every frame after the first 100: 1,948 a run.

What the runs showed

Key idea. prepare_frame and propagate_world_transforms slowed by the chain's own ratio when the thread blocked beside a busy GPU, on the same performance cluster and at priority 45 too: the clock fell; the work did not grow.

Nine runs of praxis bench <dir> --skip-aliasing town at 2560×1440 with 4x MSAA, 1,800 measured frames each, no other GPU work. Medians in µs, from the patch's log rather than praxis report, whose tables hold neither stages nor the chain. The chain's medians are with rays on, at the frame's start, after update and after render. Runs in the order they ran; the default's first run was the only one with a fast clock.

Copied from docs/performance.md; the default runs with a slow clock are shaded. The Frame stats give the CPU frame and the stages; the chain is the patch's.

What the measurements say about the cause

• The clock, not the work. The chain does the same work in every frame, so its time is the core's speed. Where it reads 2.75 µs, 1.38 to 1.42 times the spin-wait runs' mean at the same point, the prepare_frame stage and the propagate_world_transforms system read 1.39 and 1.37 times their spin-wait medians: 1,812 against 1,307 µs, and 483 against 352.
• Blocking, not the GPU's load. Spin-waiting leaves the GPU as busy, two frames in flight and the GPU frame within 40 µs, 20,119 to 20,141 against the default's 20,136 to 20,155, yet the chain stays at 1.92 to 2.04 µs.
• The wait's length and timing. With one frame in flight the thread blocks for longer, about 15 ms a frame, and the chain is slow at the frame's start, 2.25 to 2.46 µs, but fast again by the end of update, 1.92 to 1.96. The frame then runs while the GPU is idle. With two in flight it runs beside the GPU and stays slow throughout.
• Not migration. In every run, at most 1 of 1,948 frames started on an efficiency core, and at most 14 changed performance cluster between one frame's render and the next frame's start. With one exception, a few hundred changed core within a cluster, which shares its L2, so the L2 stays warm.
• Not QoS or priority. The main thread of a process started from a shell reports QOS_CLASS_USER_INTERACTIVE (0x21) from qos_class_self(), but ps -M shows it at priority 31, the default class's, whichever class a QoS call asks for. xnu's task policy has a rule that would produce the gap. For an app-type task, its switch on the task's role ends in a default case whose comment reads "Apps that don't have an application role get USER_INTERACTIVE and USER_INITIATED squashed to LEGACY" (osfmk/kern/task_policy.c, lines 879 to 883, inside the condition at 833 to 834). The patch did not read the process's app type or role, so this fits what was measured without being shown to be the cause. Then the patch tried part of what Apple recommends for a game's render thread. At WWDC18, in Metal Game Performance Optimization, Apple said: "We recommend the render set priority to be fixed to 45" (the transcript's "render set" is the render thread), and "we need to opt out of the scheduler's quality of service in order to prevent priority decay". The patch set priority 45 with pthread_setschedparam under the default time-sharing policy, SCHED_OTHER, which Apple's Tech Talk calls "compatible with QoS classes" and of which it says: "Threads using it may be subject to priority decay." It did not try the round-robin policy, SCHED_RR, whose priority is fixed. Priority 45 took effect and left the stages as slow as the default: CPU frame 6,904.2 µs, chain 2.75, 2.75 and 3.17.
• Not all of it. The first default run kept a fast clock, its chain 1.96 to 2.04 µs, yet its encode and submit read 283 and 2,710 µs against spin-waiting's 201 to 205 and 2,067 to 2,073, and its CPU frame 5,986. Blocking costs about 1 ms there that the clock does not explain. The measurements do not find what; nor does this guide.

Shipped code takes other routes. SDL gives a high-priority thread on Apple platforms the round-robin policy and priority 45; Godot and Chromium map their thread priorities and types to QoS classes (links in the reading list). On this machine a QoS call left the thread at priority 31, and priority 45 under the time-sharing policy left the clock as slow; the round-robin route was not measured.

A clock that changes run to run

The clock was fast in one run and slow in the next three, with the same binary: the first default run kept it, the three after it did not. The machine also read slower an hour later: when town's Baseline was re-recorded, the blocking build's CPU frame read 8,180 µs where it had read 6,877 to 7,279. No chain ran then, so that drift is the CPU frame's, not a measured clock. With rays off, the GPU frame is shorter but still longer than the CPU frame, and the default runs' chain reads 2.54 µs at the start, the fast first run's included. A CPU line read behind a GPU-bound frame, then, carries a clock that the work does not set, and that does not hold still.

What the bench does now

Key idea. M3's CPU ceilings price the main thread's work when it is on the critical path, so the bench spin-waits to measure that work at full clock, and the window keeps blocking, leaving the core, its energy and its heat to the frame.

M3's exit criteria hold the CPU to two ceilings: the CPU frame under 6 ms at the median and the p95 (criterion 1), and the dynamic TLAS's CPU line under 1.1 ms at the median (criterion 6), the Budgets beside them in praxis report's tables. They are about the main thread's cost when it is on the critical path, when the frame waits on the CPU. Blocked behind a GPU-bound frame it is not, and its clock falls because it has the time.

The options the owner chose from

From docs/performance.md's options, shortened.

Apple's advice against option 3 is plain, in the same Tech Talk: "Avoid busy waits. They potentially lock a P core, instead of doing something useful with it. They also prevents the scheduler from promoting a thread from and E to a P core. You are also wasting energy and producing unnecessary heat, eating away your thermal headroom." This guide infers, beyond Apple's words, that a GPU-bound game needs that headroom for its GPU. The bench is not a game: it runs for minutes, and what it must report is the work.

The code

render_frames now takes a FrameWait:

// src/capture/mod.rs at 958ed4e9.
/// How `render_frames` waits for the oldest frame in flight.
#[derive(Clone, Copy)]
pub(crate) enum FrameWait {
    /// Blocks, as a window does in `get_current_texture`.
    Block,
    /// Polls until the frame is done. […]
    Spin,
}

bench::run picks Spin, except on a software adapter:

// src/bench/mod.rs at 958ed4e9, run.
// A software rasterizer's threads would share the CPU with the spin.
let wait = match context.adapter.get_info().device_type {
    wgpu::DeviceType::Cpu => FrameWait::Block,
    _ => FrameWait::Spin,
};

• CI's lavapipe blocks. Its rasterizer runs on the CPU, where a spinning main thread would take a core from it.
• Everything else blocks too. capture and capture_frames, and so the Goldens, pass FrameWait::Block; the window never goes through render_frames and waits in get_current_texture as before.
• One Baseline moved. Only town's M4 Pro Baseline was re-recorded, the only one whose CPU lines moved: every other scene's compare passes on its CPU lines. The RTX 2070 SUPER's Baselines were recorded blocking.

After the change

The re-recorded run against the previous Baseline, from praxis report, in µs:

From docs/performance.md's "After the change", rows trimmed, the Baseline's p95 and the Δ columns left out.

The Baseline comes from a full run; two full runs read the CPU frame at 5,529 to 5,598 µs spin-waiting. An interleaved pair from one build, run with --skip-aliasing and switched between the two waits by a patch, reads it at 4,977 to 5,127. Blocking: CPU frame 8,180.0 µs, dynamic TLAS 1,567.8, GPU frame 20,473.1. Spin-waiting: 4,977.2, 943.5 and 20,759.1. That pair's GPU frame is 1.4% longer spin-waiting, where the earlier runs' were within 40 µs. docs/performance.md notes that a busy P-core may take power the GPU would otherwise have, and that judging GPU lines needs that in mind.

What is still over, and still open

• The CPU frame's p95. Criterion 1 judges the CPU frame at the median and the p95: 5,597.8 µs is under 6 ms, 6,149.2 is not. The dynamic TLAS's ceiling is judged at the median, 1,018.9, under 1.1 ms; its p95 reads 1,210.1.
• The ticket's 5%. Ticket #381 asked for stages and systems within 5% of their medians when the GPU is idle at the frame's start. Against the runs with one frame in flight, spin-waiting's prepare_frame reads 1,307 against 1,260 to 1,286 µs, within 4%, but the propagate_world_transforms system is up to 14% over and encode 21 to 24% over; submit, 2,067 to 2,073 against 2,129 to 2,569, is under. The ticket's 5% holds for prepare_frame and submit, not for propagate_world_transforms or encode.
• The remainder. The first default run's ~1 ms that its fast clock does not explain is still unexplained.

Check yourself

Reading list

Grouped by source, each with what it is good for here. Code links are pinned to 958ed4e9.

Glossary

Reference scene and Baseline are CONTEXT.md's definitions; Budget and the Frame stats tooltip are its definitions, shortened; frames in flight quotes the code. The rest are this guide's own.