A terminator that follows the normals, not the triangles

Why the terminator stepped

Key idea. The lit pass shades with the normal interpolated over each triangle, while the shadow mask judged facing by the triangle's own plane. Within a few degrees of the terminator the two disagree, and the mask hid the whole light from the whole triangle.

A coarse mesh drawn with smooth shading is flat triangles shaded with curved normals. Near the line where it turns from a light, a triangle can already face away from the light while the normals interpolated over it still face it.

The shadows sphere is a UV sphere of 48 × 24 segments, 7.5° each. A triangle's plane turns up to 5.3° from the normals interpolated over it (Shadow Rays Explained, chapter 5). Since #260 the shadow mask pass has had only the depth target to work from. It rebuilt a face normal from the depths the same sample holds at the neighbouring pixels. Where that face normal turned from a light, it set the light's bit with no ray, and elsewhere it traced a ray offset along it. The lit pass then left that light out of the pixel entirely, diffuse and specular. So in a band about 5° wide, whole triangles went dark while the shading said lit, and the terminator stepped along them.

The rebuilt face normal had a second fault. At a triangle's edge, the neighbouring pixel can show the next triangle, or another surface, so the normal rebuilt from it is neither triangle's. That darkened pixels along some wall edges on town too.

The fix that leaked

PR #258's first version tried the obvious fix: where the triangle faces away from the light while the interpolated normal faces it, skip the ray and light the pixel. The terminators came out smooth, but the helmet's smoothed bevels sit in that band, and where another part of the helmet shadowed them the rule lit them anyway. Skipping the ray ignores whatever stands in its way. The ticket listed that rule as ruled out.

A ray is still needed in the band, then. The trouble is where it starts. Offset off the triangle along its own normal, a ray toward a light the triangle turns from goes straight back through the triangle's own mesh, and finds the mesh in its way.

Three options

The ticket offered three:

1. Move the ray's origin by Hanika's terminator offset (Ray Tracing Gems II, chapter 4), which needs the shading normal, which the mask did not have.
2. Write the shading normal per sample in the depth pre-pass, a new attachment, which amends ADR 0019.
3. Offset the origin by an amount estimated from the curvature of the depth: approximate, and no new attachment.

The prototypes measured on town settled it (chapter 6). The owner chose 2b: write two normals per sample, and start the ray inside the surface where its triangle turns from the light. Option 2b uses no Hanika offset.

Two normals per sample

Key idea. The depth passes already rasterise every triangle the mask will see. With shadow rays on, they now write each sample's triangle normal and interpolated normal, so the mask reads what the triangle was instead of guessing it from neighbouring depths.

These are the Surface normals, a term #350 adds to CONTEXT.md. With shadow rays on, the depth pre-pass and phase two's depth pass both run a fragment shader, fs_surface_normals. Before, neither ran one. Their vertex stage, vs_depth_normals, passes on the lit pass's position and normal, and keeps the clip position @invariant so that their depths still meet the lit pass's exactly.

The triangle's own normal

Across a triangle, the camera-relative position changes linearly in the triangle's plane. Its screen-space derivatives, dpdx and dpdy, are two vectors in that plane, so their cross product is the plane's normal. The GPU runs fragment shaders in 2×2 quads. Where a quad hangs over the triangle's edge, the pixels outside it still run, as helper invocations for that same triangle (WGSL §15.4). So even at an edge the derivatives belong to the triangle, which is what the depth rebuild could not promise.

// main.wgsl at b8bb97f, fs_surface_normals
let crossed = cross(dpdx(in.camera_relative_position), dpdy(in.camera_relative_position));
let crossed_length_squared = dot(crossed, crossed);
let toward_camera = -normalize(in.camera_relative_position);
let unit_crossed = crossed * inverseSqrt(crossed_length_squared);
let triangle_normal = select(
    toward_camera,
    select(unit_crossed, -unit_crossed, dot(unit_crossed, toward_camera) < 0.0),
    crossed_length_squared > 0.0
);

The cross product's sign depends on how the triangle lies on screen, so the shader flips it to face the camera, the side the sample shows. A triangle seen exactly edge-on spans no plane on screen, and its normal falls back to the direction toward the camera. Neither case can produce a NaN.

The interpolated normal

The second normal is the vertex normals, turned by the Instance's normal matrix and interpolated over the triangle, then normalised: the lit pass's geometric_normal before its normal map. A normal that interpolates to zero length falls back to the triangle's. The mask reads no Textures, so normal maps play no part in it.

The target

• Format. SURFACE_NORMALS_FORMAT is Rgba8Unorm: the triangle normal in red and green, the interpolated normal in blue and alpha, each as two 8-bit octahedral coordinates (chapter 3).
• Size. 4x MSAA like the depth target, 4 bytes per sample: 2560 × 1440 × 4 × 4 = 58,982,400 bytes at 1440p, as much as the depth target itself.
• When. Written only with shadow rays on: the pre-pass clears it, and phase two's depth pass loads and stores it. With rays off the pre-pass keeps its depth-only pipeline, so the rays-off gate barely moves: +0.01 ms. The target is allocated either way, as amended ADR 0019 records.
• Per pixel. Under 4x MSAA the fragment shader runs once per pixel and its output lands in every sample the triangle covers, so both normals are pixel-centre values. The mask reads, at each pixel, the nearest sample's normals.

ADR 0019 had said its main pass writes nothing that no decided pass reads. The shadow mask is a decided pass that reads this target, and the ADR's amendment of 2026-10-28 records the new attachment, its size and its cost.

Octahedral normals in 8 bits

Key idea. A unit vector has only two degrees of freedom. Project it onto the octahedron |x| + |y| + |z| = 1 and fold the lower half over the upper, and the square [−1, 1]² holds every direction with nearly even precision. At 8 bits per coordinate, each stays within 0.95°.

Storing x, y and z wastes a number, since they always satisfy x² + y² + z² = 1. Storing x and y and rebuilding z loses the sign of z and crowds precision near the horizon. Octahedral coordinates avoid both. Divide the vector by |x| + |y| + |z| and it lands on the octahedron. Its upper half, seen from above, is the diamond |u| + |v| ≤ 1. Its lower half folds outward into the square's four corners, each triangle reflected across the diamond's edge.

// common.wgsl at b8bb97f
fn octahedral_encode(direction: vec3<f32>) -> vec2<f32> {
    let projected = direction.xy / (abs(direction.x) + abs(direction.y) + abs(direction.z));
    let sign = select(vec2<f32>(-1.0), vec2<f32>(1.0), projected >= vec2<f32>(0.0));
    return select((1.0 - abs(projected.yx)) * sign, projected, direction.z >= 0.0);
}

fn octahedral_decode(coordinates: vec2<f32>) -> vec3<f32> {
    let z = 1.0 - abs(coordinates.x) - abs(coordinates.y);
    let sign = select(vec2<f32>(-1.0), vec2<f32>(1.0), coordinates >= vec2<f32>(0.0));
    let unfolded = select((1.0 - abs(coordinates.yx)) * sign, coordinates, z >= 0.0);
    return normalize(vec3<f32>(unfolded, z));
}

fs_surface_normals maps each coordinate from [−1, 1] to [0, 1] for the Unorm target. The mask maps it back and decodes. The ticket had priced one normal as two 16-bit coordinates. Two normals at 8 bits per coordinate fill the same four bytes.

How much 8 bits lose

A new oracle test, octahedral_coordinates_decode_to_the_unit_vector_they_encode, encodes 4,110 directions on the GPU: the six axes, eight directions on the octahedron's edges, four of them on the equator where the lower half folds, and a 4,096-point Fibonacci lattice. It checks that each decodes back within 4×10⁻⁶ from full floats, and within 0.95° of itself after rounding to 8 bits. This page's own sum, in f64 with round-to-nearest, finds 0.90° over the lattice and 0.94° over 400,000 lattice directions. The bound is sampled, not proven.

Metal rounds a Unorm store to the nearest step. Vulkan only recommends it, and may round to either neighbour, which roughly doubles the error. The mask's margins in chapter 4 are sized for both.

Starting inside the surface

Key idea. Where a triangle grazes or turns from a light that its interpolated normal faces, the ray starts just inside the surface. It finds its nearest hit and passes it only if it is a back face, which means the ray is leaving its own mesh, then asks for any hit beyond. An occluder that is a closed mesh shows the ray a front face first, so its shadow holds.

A ray that starts outside a triangle turned from the light runs back into its own mesh. One that starts inside runs out of it, and the way out is the back of a triangle.

Three outcomes per light

For each light that reaches the pixel, the mask decodes the sample's two normals and picks one of three outcomes:

• Hidden, no ray, where the interpolated normal turns from the light by more than 2°: dot(interpolated_normal, direction) <= -INTERPOLATED_TURN_MARGIN, sin 2°. The 2° covers the stored normal's error, so the mask never hides a light that the lit pass's interpolated normal, its geometric_normal, still faces. Between there and the horizon, a ray is traced, and the lit pass's clamp of N·L at zero darkens what is past it.
• A ray from outside, where the triangle faces the light by at least 10°, or by at least 5° when its two normals agree within 3°, as on a flat surface: the ray starts along the triangle normal by SHADOW_RAY_OFFSET_SCALE, 2⁻¹⁶ of the point's largest coordinate and at least 2⁻¹⁶ m, as before (Shadow Rays Explained, chapter 4), and any hit hides the light.
• A ray from inside, everywhere else: the triangle faces the light by less than the margin, or turns from it, while the interpolated normal does not turn from it. The rest of this chapter is about this case.

The 5° margin is wider than the stored normals' error, for a second reason. The mask rebuilds each sample's position from depth, and that position can sit below the surface by more than the fixed ray offset. Under a grazing light, a ray leaving from there meets its own triangle. Sending those rays down the inside path hides the error, at a cost. Every flat floor that a light grazes by less than 5° traces twice. On minification, whose floor lies under grazing point lights, the mask rose from 1.05 to 3.07 ms. At a 2° margin with a 1° agreement, two pixels of the shadows sphere went wrong; an offset eight times larger cleared them. Ticket #368 will size the offset to the depth rebuild's error, after which the margin can drop toward 2°.

The inside ray

// shadow_mask.wgsl at b8bb97f, the end of shadowed()
let origin = tlas_position - triangle_normal * offset;
let exit = nearest_hit(origin, direction, distance);
if exit.kind == RAY_QUERY_INTERSECTION_NONE {
    return false;
}
if exit.front_face || exit.t + offset >= distance {
    return exit.front_face;
}
return hit_any(origin, direction, exit.t + offset, distance);

1. Start inside. The origin moves the same small offset, but into the surface, against the triangle normal.
2. Take the nearest hit. nearest_hit traces without RAY_FLAG_TERMINATE_ON_FIRST_HIT, so it returns the closest triangle rather than any. It queries the static TLAS, then the dynamic TLAS only up to the static hit, since only a nearer dynamic triangle could replace it.
3. Read its face. A ray that starts just behind a triangle facing the light crosses that triangle from behind. One that starts behind a triangle turned from the light runs through the mesh and leaves through a neighbour. Either way the nearest hit is a back face. If the nearest hit is a front face instead, the ray met something else first, and the light is hidden.
4. Look beyond. After passing the back face, a second, any-hit query starts at the exit plus the offset. Anything found hides the light: an occluder, or any other part of the same mesh.

For a point light, an exit within one offset of the light settles it: a front face there hides the light, and a back face does not. The second query would have nothing left to search. Starting it there also keeps every ray's tmin at or below its tmax, which both Metal and Vulkan require.

Why this does not leak

#203's rejected rule skipped the ray. The inside ray still runs the whole way to the light and passes exactly one triangle, the nearest, and only if that triangle shows its back. An occluder between the point and the light is a closed mesh seen from outside, so the ray enters it through a front face. If that face is the nearest hit, the light is hidden at once. If the ray first leaves its own mesh, the any-hit query beyond finds the occluder. Cast shadows fall into the band just as they fall outside it.

Front and back, per backend

A triangle's front face is the side its normal, taken from its vertices in index order, points toward. The ray query reports whether the ray met it in front_face. The Vulkan specification decides facing in the bottom-level acceleration structure's own space, so a mirrored Instance does not flip it. On Metal, MSL documents a clockwise default winding and naga sets none, which would seem to invert it; the sources do not settle it. tests/shadows.rs does: an inverted front_face would hide the sun across the terminator band it now judges, and it passes on Metal and on lavapipe. Culling back faces in the traversal with RAY_FLAG_CULL_BACK_FACING was not an option: naga's Metal backend ignores that flag. The shader reads front_face itself.

Where it can still be wrong

Key idea. The rule passes one back face and trusts that it is the ray leaving its own mesh. Where a ray starts inside something else, or a mesh is not closed, that trust can be wrong in either direction. The PR defers two checks for those cases.

• Overlapping shells. SciFiHelmet is built of shells that overlap. An inside ray that starts in one shell can meet another shell's front face first, and is hidden. One-pixel dark lines remain where the shells overlap; those pixels sat inside the dark patches before.
• Moved dots. On the helmet, 21,351 pixels brighten and 6,332 darken; on town, 1,147 and 133. The darkened ones are mostly isolated one-pixel dots of 2 to 17 steps along creases and seams. Main shows dots like them elsewhere, so the PR reads them as moved rather than multiplied, which is part of the owner's visual check.
• A possible leak. If a ray starts inside another mesh, where two meshes interpenetrate, the back face it passes may be that mesh's rather than its own. The PR defers a leak test, an occluder casting onto a terminator band, until a Golden or a visual check shows light in a cast shadow.
• Backends. The front-face convention is checked on Metal and on Vulkan through lavapipe. DX12 is unchecked while the RTX machine is stored. A probe that pins front_face per backend is deferred until DX12 runs again.
• Flat floors under grazing light. The 5° margin also covers positions rebuilt from depth that sit below the surface, at the price of a second traversal on every flat surface a light grazes (chapter 4). Ticket #368 will size the ray offset to that error instead.
• Two draws against one. With culling on, the depth passes draw the two culling phases' Instances in two draws, which write the surface normals, while the lit pass draws them in one. On the M4 Pro, which of two triangles sharing an edge covers a sample on that edge depends on what else the draw holds (Visibility Pre-pass Explained). So at a few edge samples the mask reads one triangle's normals while the lit pass shades the other. Culled and unculled town differ by up to 6 pixels a frame on Metal, and tests/goldens.rs now allows 8, HARDWARE_UNCULLED_PIXELS. Ticket #369 follows it up.
• Normal maps. The mask follows the interpolated normal, not the normal-mapped one. A bump that the normal map turns toward a light, more than 2° past where the interpolated normal turns from it, stays dark.

How it is checked

tests/shadows.rs compares the shadows scene, traced and untraced, against the true sphere's analytic shadows, intersected in f64. It used to leave out every sphere pixel within 5.3° of a terminator on its lit side, TERMINATOR_BAND. That exclusion is gone: nothing shadows the sphere, so every sphere pixel must be unchanged by the rays. On main before this change, 122 of the pixels it now judges fail.

What it costs

Key idea. On town, the fix costs the M4 Pro's GPU frame about 2.8 ms, most of it in the shadow mask, plus 58,982,400 bytes of memory. The rays-off gate barely moves. Each alternative without the triangle normal cost more, because it traced more rays, or traced them twice.

Medians from praxis report on town, on a quiet machine, run back to back with a build of main at eeb18e9:

• The frame total is the cost. Metal overlaps the mask with the lit pass, so the lit pass's own span grows by nearly as much, absorbing the mask's growth; the spans do not add. Shadow-ray time, the GPU frame with rays on less the same frame's with them off, grows by 2.70 ms. Earlier, three interleaved quiet rounds against main read +2.81 ms. The CPU frame moved by 0.12 ms, within its noise.
• Partly the fix at work. town's posts, trees, wires and pillars are smooth-shaded cylinders and cones of 6 to 24 sides: coarse, with wide terminator bands, so many of their pixels now trace the two-query inside ray where they used to be hidden with none.
• The pre-pass. Its 0.36 ms is the new fragment shader and the 4x target it writes.

The PR's A/B against main measured every Reference scene. The largest costs come from grazing light. On minification, whose flat floor the 5° margin sends down the inside path, the mask rose from 1.05 to 3.07 ms and the GPU frame by 2.0 ms. On helmet, the mask rose from 2.53 to 4.13 ms and the GPU frame by 1.6 ms. On pbr-sweep, the GPU frame rose by 0.6 ms. Ticket #368 is where those come down.

The alternatives, measured

The more precisely the mask knows which pixels are in the band, the fewer rays it traces twice. That is what the second, exact triangle normal buys.

Goldens

Six Metal Goldens moved by more than one step: town in 1,280 pixels, at wire, cone and cylinder terminators and at wall edges the depth-rebuilt normal used to darken; pbr-sweep in 546; normal-tangent in 504; shadows in 131; orrery in 52, by at most 6 steps; and uv-grid in 2. The vulkan-cpu Goldens and lavapipe Baselines came from the PR's CI, and the M4 Pro Baselines were re-recorded on a quiet machine, with gpu_memory_bytes 58,982,400 higher on every scene.

Check yourself

The mask used to rebuild a face normal from the depths at neighbouring pixels. Why was that normal wrong at a triangle's edge, and why is cross(dpdx(p), dpdy(p)) in the depth pass right there?

At an edge, the neighbouring pixel can show the next triangle, or another surface altogether, so the step to it does not lie in the sample's triangle, and neither does the normal built from it. In the depth pass, the derivatives come from the 2×2 quad the fragment shader runs in. Pixels of the quad outside the triangle still run, as helper invocations of the same triangle, with its attributes extrapolated over its plane. So both derivatives lie in that one plane, and their cross product is its normal, at the edge as in the middle.

An inside ray's nearest hit is a front face. What does the mask conclude, and why is that the safe answer?

The light is hidden. A ray that starts just inside a closed mesh should first meet the back of a triangle, where it leaves. A front face means it met something else first: an occluder it entered from outside, or another shell or mesh it started inside of. Hiding is the answer that cannot let light into a cast shadow. Its cost is the occasional dark pixel where shells overlap, as on the helmet.

Why does the mask hide a light with no ray only past −sin 2°, rather than at zero?

The interpolated normal reaches the mask through two 8-bit octahedral coordinates, within 0.95° of the lit pass's on Metal and about twice that where Vulkan rounds to either neighbour. At zero, a pixel whose stored normal had rounded just past the horizon would be hidden while the lit pass's interpolated normal still faced the light. Within the 2° margin the mask traces a ray instead, and the lit pass's clamp of N·L at zero darkens whatever is truly past the horizon.

Reading list

Grouped by source, each with what it is good for here. Links into the code are pinned to b8bb97f.

Glossary

Surface normals, Shadow mask, Reference scene and Golden are CONTEXT.md's definitions at b8bb97f. The rest are this guide's own.