Temporal Anti-Aliasing: The Idea That Changed Everything

Before DLSS could reconstruct frames, TAA taught engines to remember them.

DLSS, FSR, XeSS, and PSSR are all, at their core, temporal algorithms. They don't reconstruct an image from a single frame; they accumulate information across many frames. The technique they descend from is Temporal Anti-Aliasing (TAA), and it is the single most important rendering technique of the last decade. If you understand TAA, the rest of the course is downhill.

The aliasing problem, revisited

In the crash course we saw that a pixel is either "inside the triangle" or "outside the triangle" at its center sample. That binary decision produces jagged edges (geometric aliasing), shimmering on small features (specular aliasing), and crawling on thin lines (subpixel aliasing).

Classical anti-aliasing approaches all try to take more samples per pixel within a single frame:

SSAA (super-sampling): render at 2× or 4× resolution, downsample. Beautiful, brutally expensive.
MSAA (multi-sample): rasterize geometry edges at multiple sub-pixel positions, shade only once. Fast for forward rendering, but breaks under deferred shading, which is why MSAA died around 2014.
FXAA / SMAA: cheap edge-detection blurs applied as a post-process. Look soft, don't fix shimmer.

By the time deferred rendering and PBR became standard, the industry was stuck. MSAA was incompatible with the rendering architecture; SSAA was too expensive; FXAA was a band-aid. Something else had to give.

A 2x2 grid of close-up zoomed pixels of a thin diagonal wire against a sky, comparing four anti-aliasing techniques: no AA showing a clear staircase, FXAA with blurred edges, MSAA 4x with better edges but specular shimmer, and TAA which is smooth and slightly soft. Each labeled. Magnified pixel-art look. Clean technical comparison. — Same edge, four AA techniques only TAA combines smooth edges with shimmer suppression.

The temporal trick

The insight behind TAA: a pixel doesn't have to take its samples all at once. It can take one sample this frame, a slightly different one the next, another the frame after, and average them over time. As long as the camera and the scene don't move too fast, the result converges toward what an expensive SSAA pass would produce but spread across many cheap frames.

This is done with two ingredients:

Jittered sampling. Each frame, the projection matrix is offset by a tiny sub-pixel amount, so the "center" of every pixel falls at a slightly different position within the pixel. Over 8–16 frames, the sample positions cover the pixel area like a low-discrepancy lattice.
History reprojection. To average across frames you need to know where each pixel was in the previous frame. For that you use motion vectors: a per-pixel vector that says "this pixel was at position (x', y') last frame".

The jitter pattern is almost always a Halton sequence a deterministic low-discrepancy sequence that fills the unit square more evenly than random numbers ever could. A common choice is Halton(2, 3) with 8 or 16 unique offsets per cycle.

A single pixel shown as a big square. Inside, 16 small dots arranged in a Halton(2,3) low-discrepancy pattern, each numbered 1 to 16 in the order of sampling, connected by faint arrows showing the temporal sequence. A side note explains that each frame, the projection matrix is offset by these sub-pixel amounts. Clean diagram, dark background, monospace labels. — Halton(2,3) jitter spreads 16 sample positions evenly across a single pixel over consecutive frames.

The TAA algorithm, in pseudocode

// For every pixel of this new frame:
vec4 current = sample_current_frame(uv);                  // freshly rasterized
vec2 mv = sample_motion_vector(uv);                       // points to where this pixel WAS
vec4 history = sample_previous_frame(uv - mv);            // reproject the history

vec4 blended = mix(history, current, 0.1);                // 10% new, 90% old
write(blended);

That ~10% blend factor is the heart of it. Every frame, the pixel keeps 90% of what it had and adds 10% of the new (jittered) sample. After ~20 frames the contributions sum to the equivalent of a 20-sample super-sampled pixel. Anti-aliasing for almost free if you can trust the history.

The two failure modes

You cannot blindly trust history. Two things can ruin it:

The pixel didn't exist last frame. A new triangle appeared (something walked out from behind a wall), or the camera turned and revealed a new part of the world. This is disocclusion. The history is wrong; using it produces ghosting trails.
The pixel existed but looked different. A light turned on, a moving object changed orientation, a specular highlight moved across the surface. Using stale history produces smearing.

Real TAA spends most of its complexity handling these two cases. The standard tools are:

Velocity rejection: if the motion vector magnitude exceeds a threshold, weight history lower.
Depth rejection: if the depth at the reprojected position is wildly different from the current depth, the geometry under that pixel is different reject.
Color clamping (neighborhood clipping): look at the 3×3 neighborhood of the current frame, find the min/max color values, and clamp the history sample inside that box in YCoCg color space. If history disagrees with the local neighborhood, push it toward the neighborhood. This single trick kills most ghosting and is in every modern TAA implementation.

A 3D color cube diagram showing the YCoCg color space. Inside the cube, a small AABB box is drawn formed by the min/max of nine neighborhood samples shown as small dots. A history sample point sits outside the box. An arrow shows it being clamped or projected back onto the box surface. Labeled 'neighborhood clipping'. Clean technical diagram, dark background, isometric. — Neighborhood clipping clamps the reprojected history sample into the local 3×3 color AABB in YCoCg space.

Why TAA looks soft

TAA's reputation for being "blurry" comes from three sources:

The blend itself. Mixing old and new pixels low-pass filters detail. Sharp 1-pixel features get smeared across the blend.
Reprojection error. Motion vectors are not perfect they don't include shadow movement, refraction, or sub-pixel jitter so the history is sampled from a slightly wrong location and bilinearly filtered, which softens.
Aggressive history rejection. When TAA is conservative, it throws away valid history and falls back to a single jittered sample, which then breaks anti-aliasing.

Most modern TAA implementations apply a sharpening pass on top to compensate. Unreal Engine 4's TAA, Unreal 5's TSR, and CryEngine's TAA all have a sharpening filter built in.

TAA upscaling: where DLSS came from

Here is the punchline that makes the rest of this course possible:

Once you can accumulate sub-pixel information across frames, you don't have to be aiming at the same resolution. You can render the new frame at lower resolution, and accumulate it into a higher-resolution history buffer. Over enough frames, the history converges to a true high-resolution image.

This is TAA upsampling or TAAU. Unreal Engine added it in 4.19 (2018). It works. It is the direct ancestor of DLSS, FSR 2, XeSS, and PSSR. Every modern "AI upscaler" is, at its heart, TAAU with the hand-tuned color clamping and history validation logic replaced by a neural network that does a much better job.

A diagram showing low-resolution rasterized samples as small dots in a sparse grid being accumulated over 8 frames, each frame with a different sub-pixel jitter offset, into a higher-resolution output grid. Arrows show the temporal accumulation. Side text reads 'render 1080p, accumulate into 4K'. Clean technical, dark background. — Low-res jittered samples accumulated into a high-res history buffer the core trick behind every modern temporal upscaler.

In the next chapter we'll look at the data structure that makes all of this possible: motion vectors, what they really are, how the engine produces them, and why getting them wrong is the single biggest source of artifacts in every upscaler.