视频帧插值

Video frame interpolation (VFI) means synthesizing frames that were never filmed. Given frame A at time 0 and frame B at time 1, the goal is a believable frame at some time t in between, often t = 0.5 for doubling the frame rate. The simplest approaches either repeat a frame or blend A and B, but blending produces ghosted double images whenever something moves. Learned interpolation instead tries to work out where each object would be at time t and draw it there, which is why it can produce smooth slow motion from footage recorded at normal speed.

Most modern methods rely on optical flow, a map that gives every pixel a motion vector showing where it moves between two frames. A typical flow-based pipeline works in four steps. First, estimate flow between A and B in both directions. Second, scale and combine those flows to approximate the flow from the unknown middle frame back to each input. Third, warp, meaning resample, A and B along those flows so both line up with time t. Fourth, fuse the two warped images with weights that say which input to trust at each pixel, because some pixels are hidden in one frame and visible in the other, a problem called occlusion. A refinement network then fixes remaining errors.

NVIDIA's Super SloMo, presented at CVPR 2018, is the clearest example of this design. One U-Net, a common encoder-decoder image network, estimates the bidirectional flow; the flows are linearly combined for each time step; and a second U-Net refines the flows and predicts soft visibility maps that exclude occluded pixels before fusion. Because none of its learned parameters depend on time, it can produce any number of intermediate frames. It was trained on 1,132 clips recorded at 240 frames per second, about 300,000 frames, where real intermediate frames provide the ground truth.

A second family skips explicit flow. Kernel-based methods predict, for every output pixel, a small filter that is applied to patches of the two input frames. Niklaus, Mai and Liu's adaptive separable convolution (ICCV 2017) replaced large 2D kernels with pairs of 1D kernels, cutting the parameters needed to handle large motion and allowing a whole frame to be generated in one pass, which in turn made it practical to train with a perceptual loss. Kernel methods handle some effects that flow struggles with, but their reach is limited by kernel size.

Later work focused on speed and large motion. RIFE, accepted to ECCV 2022, estimates intermediate flow directly with a network called IFNet instead of computing two-way flow and then approximating, and its authors report it is 4 to 27 times faster than Super SloMo and DAIN while scoring better. FILM, from Google Research at ECCV 2022, targets large motion between near-duplicate photos with a multi-scale feature extractor that shares weights across scales, needs no pretrained flow or depth networks, and uses a Gram matrix loss to keep results sharp; it reports leading results on the Xiph large-motion benchmark. In September 2022 NVIDIA brought the idea to games with DLSS 3, which uses current and prior frames, hardware optical flow and engine data such as motion vectors and depth to generate a whole new frame.

The main alternatives are capture and rendering. Filming at a higher frame rate gives true intermediate frames but needs a capable camera and more light. In games, rendering more frames costs GPU time, which is why frame generation is attractive. Simple frame repetition or blending costs almost nothing but either stutters or ghosts on motion. Generative video models can also create in-between frames, and LDMVFI (AAAI 2024) explicitly frames interpolation as conditional generation with a latent diffusion model.

Interpolation fails in predictable ways. Fast or large motion breaks flow estimation and produces smearing or tearing. Occlusions, transparent or reflective surfaces, repeating textures, motion blur, scene cuts and flickering lights confuse the motion model. Thin objects such as fences and spokes often break apart. Interpolated frames also add no new information: a generated frame is a guess, so it should not be treated as evidence in forensic or scientific footage. In games, a generated frame cannot react to the newest input, which is why NVIDIA pairs DLSS 3 with its Reflex latency-reduction technology. Evaluation is also tricky: the LDMVFI authors argue that common pixel metrics such as L1 and L2 distance are poor indicators of how good an interpolated frame looks.

In practice, open-source models run on consumer GPUs. RIFE's repository reports more than 30 fps for 2x interpolation of 720p video on an RTX 2080 Ti and offers simple options such as --exp=1 for 2x and --exp=2 for 4x output. FILM's repository lists CUDA 11.2.1 and cuDNN 8.1.0 as tested versions, can split high-resolution frames into patches, and writes interpolated frames and an MP4. Training a model from scratch requires high-frame-rate video so that real middle frames can be held out as targets. DLSS 3 frame generation, by contrast, is limited by NVIDIA to RTX 40 Series hardware.