Aragtida Novel Synthesis

Novel view synthesis means producing a new image of a scene from a viewpoint where no camera stood. The input is a collection of photos, plus an estimate of where each camera was and which way it pointed, called its pose. The output is a renderer: give it any new pose and it returns the image a camera there would have taken. If a method only blends nearby photos, views far from the originals break down. Modern methods therefore build an internal 3D representation of the scene, fit it so that rendering from the known poses reproduces the input photos, and then render from new poses. What the model never saw, such as the back of a statue photographed only from the front, has to be guessed or left blank.

Almost every pipeline starts with structure-from-motion (SfM), a classical computer vision step that matches feature points across photos and solves for the camera poses and a sparse cloud of 3D points. Google's 2023 Research post on indoor reconstructions for Maps describes exactly this step: photos are processed with structure-from-motion to recover each camera's position, orientation and lens properties before any neural model is trained. Errors here, such as a wrongly placed camera, cause blur and ghosting later, because the model tries to reconcile photos that do not actually agree.

The neural radiance field, or NeRF, was introduced in March 2020 by Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan Barron, Ravi Ramamoorthi and Ren Ng, and presented as an oral at ECCV 2020. A NeRF is a fully connected neural network that takes a 3D position and a viewing direction and returns two things: a density, meaning how much that point blocks light, and a color as seen from that direction. To render a pixel, the method shoots a ray from the camera through the pixel, samples points along it, queries the network at each point and adds up the colors weighted by how much light survives to reach the camera. That process, called volume rendering, is differentiable, so the network can be trained with gradient descent simply by comparing rendered pixels with the real photos.

NeRF's results were a step change. On the paper's synthetic 360-degree scenes, built from 100 input views each, it scored 31.01 dB peak signal-to-noise ratio (PSNR, a pixel-accuracy measure where higher is better) against 26.05 dB for the best prior method, Neural Volumes. On real forward-facing scenes it scored 26.50 dB against 24.13 dB for LLFF. The whole scene fit in about 5 MB of network weights, compared with over 15 GB for LLFF on one synthetic scene. The costs were speed: training took 100,000 to 300,000 iterations, about one to two days on an NVIDIA V100 GPU, and rendering a single frame took about 30 seconds.

Three lines of work addressed those costs and limits. Instant-NGP, from Thomas Müller and colleagues at NVIDIA in January 2022 and published at SIGGRAPH 2022, replaced most of the large network with a multiresolution hash table of trainable features, cutting training to seconds or minutes and rendering 1920 by 1080 images in tens of milliseconds. Mip-NeRF 360, from Jonathan Barron and colleagues in November 2021, handled unbounded scenes where the camera circles a subject and the background stretches to the horizon, reporting 57% lower mean squared error than its predecessor mip-NeRF. Then 3D Gaussian splatting, by Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler and George Drettakis, published in ACM Transactions on Graphics in July 2023, dropped the neural network entirely. It represents the scene as millions of small, semi-transparent, stretched 3D blobs called Gaussians, each with a position, shape, opacity and view-dependent color, and draws them with a fast rasterizer instead of tracing rays.

The trade-offs show up clearly in the splatting paper's comparison on the Mip-NeRF 360 dataset. Mip-NeRF 360 reached 27.69 dB PSNR but needed 48 GPU-hours of training and rendered at 0.06 frames per second. Instant-NGP trained in under 8 minutes and rendered at about 10 frames per second, but scored 25.3 to 25.6 dB. Gaussian splatting reached 27.21 dB after 41 minutes of training, rendered at 134 frames per second on an NVIDIA A6000, and had better SSIM and LPIPS perceptual scores than Mip-NeRF 360. Its price is memory: 734 MB per scene against 8.6 MB for Mip-NeRF 360, with 1 to 5 million Gaussians in real scenes.

A separate branch tackles the hardest version of the problem, novel views from one or a few images. Zero-1-to-3, posted in March 2023 by Ruoshi Liu, Carl Vondrick and colleagues, fine-tunes a large image diffusion model on synthetic objects so that it can generate an object from a requested camera rotation given a single photo. It generalizes to real photos and even paintings, but the unseen side is invented from learned priors, not measured. Per-scene methods such as NeRF and splatting instead reconstruct only what the photos support.

Failure modes are consistent across methods. The splatting authors list artifacts in poorly observed regions, elongated or splotchy Gaussians, and popping when large Gaussians appear. NeRF-style models assume a static scene with constant lighting; Google's indoor pipeline adds per-image appearance codes from NeRF in the Wild and exposure inputs from Block-NeRF to cope with changing light, and its post lists dim bars as a hard case. Moving people, reflections, thin structures and blank walls with few features all degrade results. In practice a usable capture needs dozens to hundreds of sharp, overlapping photos with fixed exposure settings, a GPU for training, and, for splatting, a viewer that can hold hundreds of megabytes per scene.