Visuele AI-GIDS

Nieuwe weergavesynthese

Novel view synthesis is the task of rendering what a scene would look like from a camera position that was never photographed, using a set of existing photos.

Luister naar deze gidsAI-verteld · ~8 min

Op deze pagina9 minuten lezen
  1. Overzicht
  2. Belangrijkste inzichten
  3. Diepe duik
  4. Capturing and checking a small Gaussian splat scene
  5. Implementatie in de echte wereld
  6. De toekomst van nieuwe weergavesynthese
  7. Het in de praktijk brengen
  8. Bronnen en verdere literatuur
  9. Blijf verkennen
  10. Veelgestelde vragen

Overzicht

Neural radiance fields (NeRF, 2020) and 3D Gaussian splatting (2023) turned it from a research curiosity into a practical way to build explorable 3D scenes from ordinary pictures, with the newer method rendering at 1080p in real time.

Belangrijkste inzichten

  1. Novel view synthesis renders a scene from camera positions that were never photographed, using posed input photos and a fitted 3D representation.
  2. NeRF, introduced in 2020, stores a scene in a neural network queried along camera rays and scored 31.01 dB PSNR on its synthetic benchmark.
  3. Original NeRF needed one to two days of training per scene and about 30 seconds to render each frame on a V100 GPU.
  4. 3D Gaussian splatting, published in 2023, renders unbounded real scenes at about 134 frames per second with quality close to Mip-NeRF 360.
  5. Splatting trades speed for memory, using hundreds of megabytes per scene against a few megabytes for a NeRF network.
  6. Regions the input photos never saw cannot be reconstructed, only guessed, which is why single-image methods rely on generative models.

Diepe duik

Novel view synthesis means producing a new image of a scene from a viewpoint where no camera stood. The input is a collection of photos, plus an estimate of where each camera was and which way it pointed, called its pose. The output is a renderer: give it any new pose and it returns the image a camera there would have taken. If a method only blends nearby photos, views far from the originals break down. Modern methods therefore build an internal 3D representation of the scene, fit it so that rendering from the known poses reproduces the input photos, and then render from new poses. What the model never saw, such as the back of a statue photographed only from the front, has to be guessed or left blank.

Almost every pipeline starts with structure-from-motion (SfM), a classical computer vision step that matches feature points across photos and solves for the camera poses and a sparse cloud of 3D points. Google's 2023 Research post on indoor reconstructions for Maps describes exactly this step: photos are processed with structure-from-motion to recover each camera's position, orientation and lens properties before any neural model is trained. Errors here, such as a wrongly placed camera, cause blur and ghosting later, because the model tries to reconcile photos that do not actually agree.

The neural radiance field, or NeRF, was introduced in March 2020 by Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan Barron, Ravi Ramamoorthi and Ren Ng, and presented as an oral at ECCV 2020. A NeRF is a fully connected neural network that takes a 3D position and a viewing direction and returns two things: a density, meaning how much that point blocks light, and a color as seen from that direction. To render a pixel, the method shoots a ray from the camera through the pixel, samples points along it, queries the network at each point and adds up the colors weighted by how much light survives to reach the camera. That process, called volume rendering, is differentiable, so the network can be trained with gradient descent simply by comparing rendered pixels with the real photos.

NeRF's results were a step change. On the paper's synthetic 360-degree scenes, built from 100 input views each, it scored 31.01 dB peak signal-to-noise ratio (PSNR, a pixel-accuracy measure where higher is better) against 26.05 dB for the best prior method, Neural Volumes. On real forward-facing scenes it scored 26.50 dB against 24.13 dB for LLFF. The whole scene fit in about 5 MB of network weights, compared with over 15 GB for LLFF on one synthetic scene. The costs were speed: training took 100,000 to 300,000 iterations, about one to two days on an NVIDIA V100 GPU, and rendering a single frame took about 30 seconds.

Three lines of work addressed those costs and limits. Instant-NGP, from Thomas Müller and colleagues at NVIDIA in January 2022 and published at SIGGRAPH 2022, replaced most of the large network with a multiresolution hash table of trainable features, cutting training to seconds or minutes and rendering 1920 by 1080 images in tens of milliseconds. Mip-NeRF 360, from Jonathan Barron and colleagues in November 2021, handled unbounded scenes where the camera circles a subject and the background stretches to the horizon, reporting 57% lower mean squared error than its predecessor mip-NeRF. Then 3D Gaussian splatting, by Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler and George Drettakis, published in ACM Transactions on Graphics in July 2023, dropped the neural network entirely. It represents the scene as millions of small, semi-transparent, stretched 3D blobs called Gaussians, each with a position, shape, opacity and view-dependent color, and draws them with a fast rasterizer instead of tracing rays.

The trade-offs show up clearly in the splatting paper's comparison on the Mip-NeRF 360 dataset. Mip-NeRF 360 reached 27.69 dB PSNR but needed 48 GPU-hours of training and rendered at 0.06 frames per second. Instant-NGP trained in under 8 minutes and rendered at about 10 frames per second, but scored 25.3 to 25.6 dB. Gaussian splatting reached 27.21 dB after 41 minutes of training, rendered at 134 frames per second on an NVIDIA A6000, and had better SSIM and LPIPS perceptual scores than Mip-NeRF 360. Its price is memory: 734 MB per scene against 8.6 MB for Mip-NeRF 360, with 1 to 5 million Gaussians in real scenes.

A separate branch tackles the hardest version of the problem, novel views from one or a few images. Zero-1-to-3, posted in March 2023 by Ruoshi Liu, Carl Vondrick and colleagues, fine-tunes a large image diffusion model on synthetic objects so that it can generate an object from a requested camera rotation given a single photo. It generalizes to real photos and even paintings, but the unseen side is invented from learned priors, not measured. Per-scene methods such as NeRF and splatting instead reconstruct only what the photos support.

Failure modes are consistent across methods. The splatting authors list artifacts in poorly observed regions, elongated or splotchy Gaussians, and popping when large Gaussians appear. NeRF-style models assume a static scene with constant lighting; Google's indoor pipeline adds per-image appearance codes from NeRF in the Wild and exposure inputs from Block-NeRF to cope with changing light, and its post lists dim bars as a hard case. Moving people, reflections, thin structures and blank walls with few features all degrade results. In practice a usable capture needs dozens to hundreds of sharp, overlapping photos with fixed exposure settings, a GPU for training, and, for splatting, a viewer that can hold hundreds of megabytes per scene.

04Uitgewerkt voorbeeld

Capturing and checking a small Gaussian splat scene

  1. Pick a static object or room, lock the camera's exposure and white balance, and take about 100 to 200 sharp photos while walking a full circle at two heights.

  2. Run structure-from-motion software on the photos to estimate camera poses and a sparse point cloud, and discard photos that fail to register.

  3. Set aside every eighth photo as a test view that the model will never train on.

  4. Train a 3D Gaussian splatting model on the remaining photos, starting from the sparse points, for the standard 30,000 iterations.

  5. Render the held-out test views and compute PSNR against the real photos, then look at the worst images for floaters and blur.

  6. Orbit the scene in a viewer and note where artifacts appear, usually in areas that few photos covered.

Wat het laat zien

An interactive 3D scene plus a measured test score. PSNR in the mid to high 20s on real scenes is in the range the 2023 splatting paper reports; much lower scores usually mean bad poses, motion blur or changing light during capture.

Implementatie in de echte wereld

Google Maps' Immersive View shows indoor views of restaurants and cafes built from DSLR photos with a mip-NeRF 360 based model, delivered as 360-degree videos along preset flight paths, according to a 2023 Google Research post.

Turning a walk-around phone or camera capture of a room or object into an interactive 3D Gaussian splat that can be orbited at more than 30 frames per second at 1080p, the target set by the 2023 splatting paper.

Training a radiance field in minutes rather than days with NVIDIA's Instant-NGP, whose 2022 paper reports high-quality results after a matter of seconds of training using fused CUDA kernels.

Generating plausible side and back views of an object from a single photo with Zero-1-to-3, a 2023 diffusion model conditioned on a relative camera rotation, as a starting point for single-image 3D reconstruction.

Comparing new methods on the Mip-NeRF 360 dataset of unbounded real scenes, the benchmark where 3D Gaussian splatting reported 27.21 dB PSNR against 27.69 dB for Mip-NeRF 360.

De toekomst van nieuwe weergavesynthese

Several open problems stand out in the primary literature. The 2023 Gaussian splatting paper names memory as a cost, with hundreds of megabytes per scene and training peaks above 20 GB for large scenes, so compression and streaming are active topics. It also lists artifacts in poorly observed regions and popping during camera motion, which points to better regularization and visibility handling. Google's 2023 Research post on indoor reconstructions names real-time on-device neural rendering, scene segmentation for search and navigation, and outdoor photo collections as open questions, and it chose pre-rendered videos over live rendering because real-time inference was too costly for phones. Dynamic scenes with moving people and changing light remain harder than static ones. Few-view and single-view synthesis, illustrated by Zero-1-to-3 in 2023, depends on generative priors that invent unseen content, so measuring whether generated views are faithful rather than merely plausible is still unsettled. Whether radiance fields, splats or generative models will dominate particular applications is an open question.

Het in de praktijk brengen

Waarom het ertoe doet

  • Snelheid en schaal. Visuele AI kan inspectie-, detectie- en taggingtaken op schaal automatiseren.
  • Bouwkeuzes. Creatieve teams kunnen concepten sneller prototypen met minder handmatige revisies.
  • Team en workflow. Bij bewerkingen kan gebruik worden gemaakt van beeld- en videosignalen die voorheen moeilijk te verwerken waren.

Pas op voor

  • Beeldrechten en toestemming kunnen juridische risico's worden als de herkomst onduidelijk is.
  • De prestaties van modellen kunnen variëren afhankelijk van de belichting, demografische gegevens en omgevingen.
  • Valse positieve resultaten kunnen onopgemerkt blijven, tenzij de vertrouwensdrempels worden gecontroleerd.

Volgende stappen

  1. Definieer acceptatiecriteria voor precisie-, terugroep- en foutkosten.
  2. Test met gegevens die overeenkomen met echte productieomstandigheden.
  3. Voeg menselijke beoordeling toe voor voorspellingen met weinig vertrouwen of hoge impact.
  4. Volg modelafwijkingen en valideer opnieuw na wijzigingen in de camera of dataset.

Bronnen en verdere literatuur

  1. arXiv (ECCV 2020)NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis
  2. arXiv (ACM Transactions on Graphics, 2023)3D Gaussian Splatting for Real-Time Radiance Field Rendering
  3. arXiv (SIGGRAPH 2022)Instant Neural Graphics Primitives with a Multiresolution Hash Encoding
  4. arXivMip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields
  5. arXivZero-1-to-3: Zero-shot One Image to 3D Object
  6. Google ResearchReconstructing indoor spaces with NeRF

Blijf verkennen

Gratis nieuwsbrief

Ontvang de dagelijkse AI-briefing

Drie verifieerde AI-verhalen elke weekdagmorgen, geschreven in eenvoudig Engels. Gratis voor altijd, geen advertenties.

Eén e-mail elke weekdag. Afmelden met één klik. We verkopen of delen je adres nooit.

Test jezelf

Take the Novel View Synthesis quiz

Directe feedback op elke antwoord, en een deelbare certificaat met een verifieerbare ID zodra je een cursus hebt afgerond.

Quiz starten

Ondersteun gratis AI-onderwijs. AI Understanding is een 501(c)(3) non-profit — geen advertenties, geen betaalmuur, nooit. Doe een donatie

Veelgestelde vragen

Wat is de nieuwe weergavesynthese?

Novel view synthesis is the task of rendering what a scene would look like from a camera position that was never photographed, using a set of existing photos. Neural radiance fields (NeRF, 2020) and 3D Gaussian splatting (2023) turned it from a research curiosity into a practical way to build explorable 3D scenes from ordinary pictures, with the newer method rendering at 1080p in real time.

Wat probeert de nieuwe weergavesynthese, gegeven foto's van een scène, te produceren?

Nieuwe weergavesynthese produceert fotorealistische beelden vanuit nieuwe, onzichtbare gezichtspunten met behulp van een reeks invoerbeelden met bekende poses.

Waarop brengt een Neural Radiance Field (NeRF) een 3D-positie en kijkrichting in kaart?

NeRF leert een functie van (positie, richting) naar uitgezonden kleur en dichtheid, die vervolgens langs stralen wordt geïntegreerd om afbeeldingen weer te geven.

Hoe rendert NeRF een pixel voor een nieuwe weergave?

Voor elke pixel werpt NeRF een straal uit, bemonstert 3D-punten, vraagt ​​het netwerk naar kleur/dichtheid en stelt deze samen met de volume-rendering-integraal.

What distinguishes 3D Gaussian Splatting's scene representation from an original NeRF-style per-ray network evaluation?

The representations and rendering computations differ; speed and memory depend on their implementation and scene.

Waarom kunnen NVS-methoden reflecties en glanzende highlights reproduceren?

Door kleur afhankelijk te maken van de kijkrichting, vangen NeRF- en Gaussiaanse splats weergave-afhankelijke effecten op, zoals spiegelende hooglichten en reflecties.